CISPA

CISPA

Bookmark and Share
Posted in Uncategorized | Comments Off

BradLucas.com

Getting back to things and have updated my personal page at bradlucas.com.

It is a simple page with six thumbnails. What made the update easy was I decided to use Twitter Bootstrap (http://twitter.github.com/bootstrap/). What made it nice made it responsive so it works well on any device.

Bookmark and Share
Posted in Beacon Hill, Inc. | Comments Off

Building Command Line Applications with Clojure

To be able to package your Clojure application as a single jar and to run it from a script or the command line is a very handy thing. At first it might not be all that apparent how to do this if you are staring to learn Clojure and are happily hacking away in the Repl or evaluating code from Emacs using Slime. But, it is possible and very easy to do. You’ll need to be using Leiningen so at this point I’ll assume you are.

Java and One-JAR

Back in the day when you wanted to deliver your Java application you had to collect all the dependent Jar files and your own Jar or class files and then come up with a script to kick the whole thing off. You had to make sure things were all in the right place and your Classpath was correct.

Then On-JAR came along. At the time I thought this was fantastic. What it did was provide an Ant task that could package up your code with all the dependent files into a single jar file. You ran your application like so; “java -jar your-app-cmdline.jar”. I used it quite often for a number of applications.

Leiningen Uberjar

Clojure, being built on Java would have similar issues if you wanted to package things up yourself and deliver class files and Jars. Nicely, the build tool Leiningen has the ‘uberjar’ command. This packages your app with it’s dependencies into a single stand alone jar just like One-JAR. To enable this you only need to do a few things in your project.

1. Edit project.clj

First, you need to edit your ‘project.clj’ file and add key :main with a value that is your main namespace.

(defproject cmdline "1.0.0"
  :description "Command line application in Clojure"
  :dependencies [[org.clojure/clojure "1.3.0"]]
  :main cmdline.core
  )

2. Edit core.clj

Next, go into the namespace file you just put in project.clj and add a gen-class form like the following.

(ns cmdline.core
  (:require [clojure.java.io :as io])
  (:gen-class :main true))

3. Define -main

Lastly, you need to define a -main function. The :gen-class will tell clojure to build a class files with a ‘main’ function. The -main you define will be main function.

(defn -main
  "The application's main function"
  [& args]
  (println args))

Build and Run

With those simple three steps completed you use Leiningen to build your single jar.

$ lein uberjar

Then you run your application as follows. Any arguments will be presented to your -main function as a list in the args variable.

$ java -jar cmdline-1.0.0-standalone.jar hello!

Parameters

You’ll notice that your function -main is defined to take an optional value args which will contain any arguments you sent on the command line. For simple applications you can process this list by hand. For example, suppose you expect a single value which you’ll use to do something with. If this value is present you’ll print out a usage statement. This can be done like so.

(defn -main
  "The application's main function"
  [& args]
  (if args
    (println (str "You passed in this value: " args))
    (println "Usage: cmdline VALUE")))

For simple scenarios this works but quickly becomes unmanageable when you start thinking of flags and multiple parameters.

Tools.cli

For more complex command line parameter scenarios there is a library called tools.cli. Available on github at ttps://github.com/clojure/tools.cli, the library supports command line parameter parsing in a simple fashion.

To use the library in your project you only need to add “[org.clojure/tools.cli "0.2.1"]” to your project file in the :dependencies section.

(defproject cmdline "1.0.0"
  :description "Command line application in Clojure"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/tools.cli "0.2.1"]]
  :main cmdline.core
  )

Then you need to reference it in your namespace declaration.

(ns cmdline.core
  (:use [clojure.tools.cli :only (cli)])
  (:gen-class :main true))

cli

The only function you need is cli. You pass your args variable to cli along with vectors that describe the parameters you are looking for. These vectors are descriptors for each parameter the application is going to handle.

For example, if you want to support a help flag. You might want this flag to be ‘-h’ or ‘–help’ and if either is present have the app respond with some usage text. In this case you’ll want the app to not be bothered if the help flag is not present. The vector syntax for this would be the following:

["-h" "--help" "Show help" :flag true :default false]

Another example, would be a parameter which you expect to be a number. Also, you want the app to set a value to a default value if this parameter is not present. The following is syntax for a delay parameter which if not present defaults to 2.

["-d" "--delay" "Delay between messages (seconds)" :default 2]

If you want to specify a simple value you can do that as follows.

["-f" "--from" "From address"]

With a set of these vectors you pass them along with the args value to cli which returns three vectors. The first is a options map with keys that match you specified parameter names and their associated values. The second vector is the original args value and the last is a value containing the usage text which when printed explains the parameters you’ve declared.

If you find a required value missing from your options map you can simply print the banner and exit.

Example For Command Line Email Application

The following is a real-world example from an application that mass mails a email to a list of recipients. Notice, that is supports a help flag, has a few defaulted optional parameters (delay and test) as well as few parameters that expect values (from, email-file, subject and message-file).

The other thing to notice is how the first parameter returns from cli, here called opts to indicate these are the options is a map and you can get the values by using the keys you described in the initial call to cli. For example, you can get the subject value with ‘(:subject opts)’.

The run function included here simply prints out the options and arguments passed to it.

(defn run
  "Print out the options and the arguments"
  [opts args]
  (println (str "Options:\n" opts "\n\n"))
  (println (str "Arguments:\n" args "\n\n")))

(defn -main [& args]
  (let [[opts args banner]
        (cli args
             ["-h" "--help" "Show help" :flag true :default false]
             ["-d" "--delay" "Delay between messages (seconds)" :default 2]
             ["-f" "--from" "REQUIRED: From address)"]
             ["-e" "--email-file" "REQUIRED: Email addresses FILE)"]
             ["-s" "--subject" "REQUIRED: Message subject"]
             ["-m" "--message-file" "REQUIRED: Message FILE"]
             ["-b" "--bcc" "BCC address"] ;; optional
             ["-t" "--test" "Test mode does not send" :flag true :default false]
             )]
    (when (:help opts)
      (println banner)
      (System/exit 0))
    (if
        (and
         (:from opts)
         (:email-file opts)
         (:subject opts)
         (:message-file opts))
      (do
        (println "")
        (run opts args))
      (println banner))))

Example Code

The GitHub repository for this example is available at http://github.com/bradlucas/cmdline.

Bookmark and Share
Posted in Clojure, Programming | Tagged | 3 Comments

Loading Data with Clojure

If you are coming to Clojure from Java you are probably familiar with JDBC. If not do a bit of research and you’ll find that it is a layer that allows you to program against a database agnostic layer that communicates to your database through a database specific library.

In the following example we’ll use the clojure.java.jdbc wrapper for JDBC-based access to databases. This library is not part of what you would automatically have when you just install Clojure. For use to use it we’ll have to download it ourselves or use a build tool to do it for us. To keep things simple here I’ll assume you are using Leiningen as a build tool. If not you can find clojure.java.jdbc on GitHub.

Also, you will have to have a database. Here I’ll assume MySQL and explain how to connect to that.

The example we’ll build here will be an enhancement to the stock quote downloader presented in Reading Files article. Instead of writing the quotes to a file we’ll load them into a database. You might want to review the article before continuing.

Database

Here we’ll use MySQL but you could be using another. The only difference in the following will be the setting up of your database’s JDBC library.

Install and Setup JDBC Driver

You’ll need to install and reference the MySQL JDBC driver. At this point you should consider using a build tool. Manually you’ll need to download the jar file and make sure it is on your class path before starting your Clojure Repl or running Clojure to evaluate your file. Both techniques are worth understanding thoroughly but once you do understand you’ll find using a build tool to be more efficient and satisfying.

Manual Install

You’ll need to download a copy of the JDBC Driver for MySQL. You can do that on the MySQL Connectors page (http://www.mysql.com/products/connector/). Grab the one for Java. They call it Connector/J. (Download link http://www.mysql.com/downloads/connector/j/)

Inside the package the file you want is the jar named mysql-connector-java-5.1.18-bin.jar. Before you run your Clojure repl you’ll want this file on your Classpath.

Using a Build Tool

There are two popular build tools for Clojure, Leiningen and Cake. They are similar in idea to Ant in that they help manage where dependencies are, which files are source files and where to put compiled files. Make sure you understand using Clojure from the command line and with a command-line started Repl before using a build tool so you have a deeper understanding of what the build tool is solving.

I use Leiningen and there is talk of a merging with Cake so let’s go with Leiningen here.

(At some point I’ll write an intro to Leiningen. Here I’ll assume you figured it out and you have installed Leiningen and can create a new project)

After you’ve created your project with “lein new quote-loader” you’ll see a project.clj file. Here is where you put your external dependencies. In this case we need the MySql driver. To include it you add “[mysql/mysql-connector-java "5.1.18"]” to the dependencies vector. Notice also that you need to add [org.clojure/java.jdbc "0.1.1"] to get the clojure.java.jdbc support.

(defproject quote-loader "1.0.0"
  :description "quote-loader"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/java.jdbc "0.1.1"]
                 [mysql/mysql-connector-java "5.1.18"]])

Analyze Quote Data

We previously downloaded our quote data so we know what the data looks like. The first line is a header and it tells us what the numbers mean. We will want to ignore this line when processing the file.

Date,Open,High,Low,Close,Volume,Adj Close

Subsequent rows are the quotes each on their own line.

2012-02-17,31.20,31.32,30.95,31.25,70036500,31.25

Data is present in a descending order. The oldest is last in the file.

Create Database and Table

First let’s create database called quote. Then create a test user with access to that database.

mysql> create database quote;

mysql> create user 'tester'@'localhost' identified by 'password';

mysql> grant all on quote.* to 'tester'@'localhost';

Next, login as ‘tester’ and create the quote table.

drop table if exists quote;
create table quote (
       quote_id    int(10) unsigned not null auto_increment,
       symbol      varchar(6),
       date        date,
       open        decimal(18,4),
       high        decimal(18,4),
       low         decimal(18,4),
       close       decimal(18,4),
       vol         bigint,
       adjclose    decimal(18,4),
       primary key (quote_id),
       index (symbol, date)
);

-- YYYY-MM-DD  http://dev.mysql.com/doc/refman/5.1/en/datetime.html
-- http://stackoverflow.com/questions/1523576/database-schema-for-organizing-historical-stock-data

Parsing Quote Data

Revisiting our sample data we’ll have to have routine to parse the comma delimited fields so we can load them into the appropriate database columns.

2012-02-17,31.20,31.32,30.95,31.25,70036500,31.25

Parsing CSV data

To help us parse each line will use David Santiago’s clojure-csv library. This is another external library we’ll want our build tool to manage for us. The following will need be included in our dependencies list in the Leiningen project.clj file.

[clojure-csv/clojure-csv "1.3.2"]

To use the library require it as follows. Note I’ve given it a shortened name of csv to make the code that uses the library more readable.

(require '[clojure-csv.core :as csv])

Now we can call csv/parse-csv on each row to get a vector with items for each value in our row. The call returns a vector of strings with a vector for each row. It is meant to handle an entire file but here I’m using on a single row. With that the first call is needed to pull the first row out. If you want to try this by hand enter the following.

(def test-data "2012-02-17,31.20,31.32,30.95,31.25,70036500,31.25")

(first (csv/parse-csv test-data))

You should get back the following.

["2012-02-17" "31.20" "31.32" "30.95" "31.25" "70036500" "31.25"]

Now, we’ll need those values in some sort of structure that allows use to put them in the database in the appropriate columns. We could do this position-ally but that wouldn’t be very clear if we revisited the code in the future. What would be nice is to convert the vector to a map so we could pull values out by keywords.

To do that we can use the zipmap function that returns a map with the keys mapped to the corresponding values. The following shows that in a parse-row function.

(defn parse-row [row]
  (let [v (first (csv/parse-csv row))]
    (zipmap [:date :open :high :low :close :vol :adjclose] v)))

Try calling parse-row with the test-data as a parameter to see how it works.

user=> (parse-row test-data)

You should get the following.

{:adjclose "31.25", :vol "70036500", :close "31.25", :low "30.95", :high "31.32", :open "31.20", :date "2012-02-17"}

Connecting to the Database

Over on wikibooks where is a short example showing how to connect to MySQL. You can review it or use the following as an example.


(def db-host "localhost")
(def db-port 8889)
(def db-name "quote")
(def db-user "tester")
(def db-pwd "password")

(def database-dev-settings
  {
   :classname "com.mysql.jdbc.Driver" ; must be in classpath
   :subprotocol "mysql"
   :subname (str "//" db-host ":" db-port "/" db-name)
   :user db-user
   :password db-pwd
  }
)

Inserting into the database

Much like when you are reading or writing a file and you need to do so within a ‘with-open’ call our database calls will be done within a ‘with-connection’ call. The ‘with-connection’ call takes our database-settings hash and connects to the database executing the code we pass to it as a parameter. In our example we’ll use update-or-insert-values. This is nice because we can rerun our loader everyday and it will update previous values and insert only the new values. Certainly not the most efficient plan but a simple one.

(defn insert-quote
  "Insert or update a quote passed in as a map"
  [sym q]
    (sql/with-connection database-dev-settings
      (sql/update-or-insert-values
       :quote
       ["symbol=? and date=?" sym (:date q)]
       {:symbol sym :date (:date q) :open (:open q) :high (:high q) :low (:low q) :close (:close q) :vol (:vol q) :adjclose (:adjclose q)})))

Using the assoc function

Based on some initial comments to this article we can rework insert-quote a bit to more readable. The think to look at is the map that is created and passed into update-or-insert-values. The line in question is.

{:symbol sym :date (:date q) :open (:open q) :high (:high q) :low (:low q) :close (:close q) :vol (:vol q) :adjclose (:adjclose q)}

First, realize that we are creating a new map from our parameters ‘sym’ a string and q a map. Also, notice this new map has the same keys as our parameter map q with the additional new key or :symbol. This and the function assoc gives a more readable method for creating a new map.

(assoc q :symbol sym)

The assoc function creates a new map with the same key values as the parameter q with the addtional key value of :symbol sym. Exactly what we were doing more verbosely in the earlier version. Using this new method we can improve insert-quote as follows.

(defn insert-quote1
  [sym q]
    (sql/with-connection database-dev-settings
      (sql/update-or-insert-values
       :quote
       ["symbol=? and date=?" sym (:date q)]
         (assoc q :symbol sym))))

Destructuring

There is another technique we can use to create our map to pass into update-or-insert-values, destructuring. Destructuring, also called abstract structural binding lets you pass variables in side of let binding lists and function parameter lists. It results in more concise code because assignments from data structures being passed in is done before the body of the let or function. This technique can be used in the insert-quote example as follows. Notice how the q map as a parameter is being destructured into variables which will be used inside the function.

(defn insert-quote2
  [sym {date :date
        open :open
        high :high
        low :low
        close :close
        vol :vol
        adjclose :adjclose}]
  (sql/with-connection database-dev-settings
    (sql/update-or-insert-values
     :quote
     ["symbol=? and date=?" sym date]
    {:symbol sym :date date :open open :high high :low low :close close :vol vol :adjclose adjclose})))

(see also {http://blog.jayfields.com/2010/07/clojure-destructuring.html)

Load Historical Quotes

The last thing we need to remember before we write our load function is that we need to ignore the first line. We can do this by knowing which line we are on but another way is to check the line’s content before we try for the insert. A simple test is to see if the first character of the line is a number. We know that the header contains letters and our data is numbers so this is reasonable. We could enhance this to check the values in the line as well. A validation function but let’s leave that for now.

(defn valid-data [str]
  (Character/isDigit (first str)))

(defn load-historical-quotes [sym]
  (let [url (build-url sym)]
    (with-open [rdr (io/reader url)]
      (doseq [line (line-seq rdr)]
        (if (valid-data line)
          (insert-quote sym (parse-row line)))))))

Source Code

The following is the entire example in one file. The project.clj file is shown first to show the required dependencies then the clj file, quote-loader.clj. If you create a new project with ‘lein new quote-loader’ then replace your project.clj and core.clj file you’ll have everything you need.

Remember to edit the db-user and db-pwd values (DB-USER and DB-PASSWORD for your database.

(defproject quote-loader "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [clojure-csv/clojure-csv "1.3.2"]
                 [org.clojure/java.jdbc "0.1.1"]
                 [mysql/mysql-connector-java "5.1.18"]
                 ])

(ns quote-loader.core
  (:require [clojure.java.io :as io])
  (:require [clojure.java.jdbc :as sql])
  (:require [clojure.string :as str])
  (:require [clojure-csv.core :as csv]))

(defn build-url [sym]
  (str "http://ichart.finance.yahoo.com/table.csv?s=" sym "&ignore=.csv"))

(def db-host "localhost")
(def db-port 8889)
(def db-name "quote")
(def db-user "tester")
(def db-pwd "password")

(def database-dev-settings
  {
   :classname "com.mysql.jdbc.Driver" ; must be in classpath
   :subprotocol "mysql"
   :subname (str "//" db-host ":" db-port "/" db-name)
   :user db-user
   :password db-pwd
  }
)

(defn parse-row [row]
  (let [v (first (csv/parse-csv row))]
    (zipmap [:date :open :high :low :close :vol :adjclose] v)))

(defn valid-data [str]
  (Character/isDigit (first str)))

(defn insert-quote
  "Insert or update a quote passed in as a map"
  [sym q]
    (sql/with-connection database-dev-settings
      (sql/update-or-insert-values
       :quote
       ["symbol=? and date=?" sym (:date q)]
       {:symbol sym :date (:date q) :open (:open q) :high (:high q) :low (:low q) :close (:close q) :vol (:vol q) :adjclose (:adjclose q)})))

(defn load-historical-quotes [sym]
  (let [url (build-url sym)]
    (with-open [rdr (io/reader url)]
      (doseq [line (line-seq rdr)]
        (if (valid-data line)
          (insert-quote sym (parse-row line)))))))

GitHub

See https://github.com/bradlucas/quote-loader for a working version of the example presented here.

As a bonus the uploaded project includes support to run as a standalone program from the command line.

Bookmark and Share
Posted in Clojure, Programming | Tagged | 4 Comments

Writing Files in Clojure

In the previous post we saw how to read files and then introduced a function to write them. The example presented downloaded stock quotes from Yahoo and then writes the data to a local file. Let’s look a bit more into writing to files in Clojure.

Writing Short Strings

The opposite of slurp is spit. This is for writing short strings to files. You call the function with the name of the file and the string.

user=> (spit "testfile.txt" "This is my string")

After running the example you should see a file called ‘testfile.txt’ with the contents of ‘This is my string’.

Writing Large Files

To write larger amounts of data we’ll need to use a writer. To review, we can use clojure.io.writer to get a java.io.BufferedWriter. This is used within a with-open call.
Suppose for the purposes of demonstration we want to create a file that is simply a list of sequencial numbers from 1 to 999 with each number on it’s own line. The first thing we’ll need is the list of numbers from 1 to 999. This can be gathered with the range function. For example here are the numbers from 1 to 10.

user=> (range 1 10)

We’ll need to iterate over the list so lets print them as an example.

user=> (doseq [x (range 1 10)]
          (println x))

Now, we just need to do the same but writing to a file. At this point you can review the second part of the reading files article which ends with an example that writes to a file.

Code

First we’ll need a writer from the clojure.java.io namespace so we enter the following statement to setup clojure.java.io as io.

(require '[clojure.java.io :as io])

Next, we’ll write our ‘write-thousand-lines function as follows. In the function we wrap using the with-open connect our writer to a file and keep the connection while we doseq over our range of numbers writing each value to the file on it’s own line. Note the Dot special form that calls the write member function of or writer accepting our string.
How would you know that? First you have to realize that the writer is a java.io.Writer which you can in the doc for the function (see writer). Then you have to know what a java.io.Writer is from when you used to work in Java. If you don’t here is the doc. There you can see the write method that you need to call.

(defn write-thousand-lines [filename]
  (with-open [wrt (io/writer filename)]
    (doseq [x (range 1 1000)]
      (.write wrt (str x "\n")))))

Finally, we can create our thousand line file with the following statement.

(write-thousand-lines "thousand.txt")

Look for ‘thousand.txt in your current directory.

GitHub

See https://github.com/bradlucas/quote-downloader for a working Clojure program that demonstrates reading data from a url and writing to a local file.

As a bonus the project includes support to run as a standalone program from the command line.

See the previous posts Reading Files Part 1 and Reading Files Part 2.

Next

Loading data with Clojure. Available soon.

Bookmark and Share
Posted in Clojure, Clojure, Programming | Tagged | 1 Comment

Yahoo Quote Downloader in Clojure

Yahoo provides downloadable historical quote data if you exercise a properly formatted URL. The data is returned in CSV format and is easily stored in a file. I wrote quote-downloader to accept stock symbols from the command line and request and store the data in symbol.csv files locally.

The program demonstrates reading from the command line, building a unique URL and reading from it as well as saving the results to a file and

The code is on github here
https://github.com/bradlucas/quote-downloader

Bookmark and Share
Posted in Clojure, Clojure, Programming | Tagged | 1 Comment

Reading Files in Clojure – Part 2

In the previous part we saw how to read files and print them. This isn’t terribly interesting. Typically, you are going to read files and then do some processing on the contents of the file and possibly save the results. Let’s do that in a set of examples.

Functionally

Clojure is a functional language and you’ll realize that as you learn the language and start writing functions. There is actually more to it than that but for now remember that you are building your program by creating and composing functions. In a purely functional language each function would not have a side effect. In other words the function would accept parameters and return a result without any change to the environment around it. As soon as we started talking about writing a function to process a file and do something with the contents we introduced the idea of a side effect.

In our simple example previously we printed the lines. Next we’ll process the lines and write the results to another file.

File Copy and Upper Case Example

How about for a totally contrived example we write a function that takes a file, reads and write a copy where everything is uppercased. We know how to read a file let’s learn how to write one.

If we return to the clojure.java.io namespace we’ll find there is a writer that wraps a java.io.BufferedWriter. This should work.

The following example makes a copy of .bashrc as FOO. A couple things to note. First we need to call the write method on our writer. See that java.io.Writer doc to verify this. Also, see the dot macro in action. To call the write method on our wrt instance we use .write. See Java Interop page on Clojure.org page for more details.


(with-open [rdr (io/reader "/Users/brad/.bashrc")
            wrt (io/writer "/Users/brad/FOO")]
    (doseq [line (line-seq rdr)]
       (.write wrt (str line "\n"))))

Now we need to know how to upper case a string. We’ll do this to each string before we write it. The function we’ll use is \link{href{http://clojuredocs.org/clojure_core/clojure.string/upper-case}\text{upper-case}} in the clojure.string namespace. To access it use the following in our repl.


(use '[clojure.string :as str])

And as a test try the following.


user=> (str/upper-case "this is lower case")

And finally finish up our file copy upper cased routine.

(with-open [rdr (io/reader "/Users/brad/.bashrc")
            wrt (io/writer "/Users/brad/FOO")]
  (doseq [line (line-seq rdr)]
    (.write wrt (str (str/upper-case line) "\n"))))

Reading URLs

Reading local files is nice but there is a whole world out on the Internet which is readable as well. Nicely, the reader function can read from URLs so we can slurp or read any URL we have access to.

So as a quick example read my web site.

(slurp "http://beaconhill.com")

Slurp is fine for quick things like this but what if we want something a bit more functional. What if for example we want to download stock quotes from Yahoo.

For this we need to know the URL format to get all the stock quotes for a given symbol. I happen to know it looks like this.

http://ichart.finance.yahoo.com/table.csv?s=SYMBOL&ignore=.csv

So we’ll need a function to build this URL given a stock symbol and then a function to read that URL into a local file. Say we’ll save the data to a file with the SYMBOL as it’s name.

Building Functions

Before now we’ve written our routines and run them in the repl. At this point you should learn how to build functions so you can call them from other functions. To do this you’ll with a macro called defn. This takes a name and a function definition creates the function and assigns it to your name so you can call it later.

Let’s start with a function to build a url given a symbol.

(defn build-url [sym]
  (str "http://ichart.finance.yahoo.com/table.csv?s=" sym "&ignore=.csv"))

Notice how we can use the str function simply concatenate the parts of the URL we need with the symbol parameter.

Next, we should take our earlier example and modify it read the url and write the data to a local file. Let’s call the data file SYMBOL.csv.

Let’s assume our data can be large so slurp is not a good idea. We’ll use our reader writer example.


(defn download-historical-quotes [sym]
  (let [url (build-url sym)
        filename (str sym ".csv")]
    (with-open [rdr (io/reader url)
                wrt (io/writer filename)]
      (doseq [line (line-seq rdr)]
        (.write wrt (str line "\n"))))))

In this example, our let block sets up two variables, the url to read from and the filename to write to. The url is created using our build-url function and we create filename by appending .csv to our symbol. Then we create our reader and writers inside a with-open as before. With those we we call line-seq as before on our reader and doseq across the sequence to get each line. Each line is then written using our writer which writes the string to our file.

Try the following.

(download-historical-quotes "goog")

You should see a goog.csv file in your local directory containing all of the vailable Google historical quote data.

GitHub

See https://github.com/bradlucas/quote-downloader for a working version of the example presented here.

As a bonus the uploaded project includes support to run as a standalone program from the command line.

Next

Writing files in Clojure


See also: http://www.beaconhill.com/

Bookmark and Share
Posted in Clojure, Programming | Tagged , , , | 2 Comments

Reading Files in Clojure – Part 1

There are a few ways to read files in Clojure.

slurp

If you want to simply read a small file into a string then you can use slurp. This function works with local files as well as URLs.

An example:


user=> (print (slurp "/Users/brad/.bashrc"))


clojure.java.io

The clojure.java.io namespace contains a reader function that returns a java.io.Reader. Make sure to call this inside a with-open call.

To use clojure.java.io place the following namespace declaration at the top of your file.

(ns your-namespace-name
   (:require [clojure.java.io :as io]))

If you are in the repl and in user you can just require clojure.java.io like this.

(require '[clojure.java.io :as io])

Now as an example you can read a file and print it line by line.

user=> (with-open [rdr (io/reader "/Users/brad/.bashrc")]
         (doseq [line (line-seq rdr)]
            (println line)))

Lazy

The thing to notice in this example is that the code (print line) that is using the contents of the file is inside a with-open call. Also, the reader is being passed to the line-seq call, the return from which is a lazy sequence of strings from the file.

This lazy sequence concept is important to understand when working with Clojure. One way to look at is by considering what it is not. The lazy sequence is not an array which holds the entire set of strings but rather a mechanism that can get you the strings in a sequence as you need them.

In this example the doseq is doing the iterating by pulling each subsequent line out of the line-seq into line.

So, if you use this example you must realize that what you want to do to each line must be inside the with-open so the file stays open and with a sequence function like doseq to iterate over the sequence. This might seem foreign to readers expecting to call a function to get all the lines in a file, then iterate over them and then pass each line to a function. Here we must think functionally.

doall

If you must have the entire set of strings from the file returned from your function you must realize the entire lazy sequence. The doall is used to do this.

In the following example you see doall is passed the line-seq results. Here doall will iterate through the entire sequence and return the fully realized sequence. Try the following and you see a list of strings returned.

(with-open [rdr (io/reader "/Users/brad/.bashrc")]
    (doall (line-seq rdr)))

Now, suppose you want to print those lists. See how you can again look through the sequence and print them.

(doseq [line (with-open [rdr (io/reader "/Users/brad/.bashrc")]
    (doall (line-seq rdr)))] (println line))))

Part 2

View part 2 here


See also: http://www.beaconhill.com/

Bookmark and Share
Posted in Clojure, Programming | Tagged , | 3 Comments

S3cp now on GitHub

To make it easier for others to work on I’ve moved my most popular open source project to GitHub.

Already, I’ve had one pull request with some fixes to allow s3cp to run under Windows more easily.

See the project at https://github.com/bradlucas/s3cp

Bookmark and Share
Posted in Java, Open Source | Comments Off

Error NT_STATUS_BAD_NETWORK_NAME printing from Mac to Linux Printer

Just fixed. Was getting

Tree connect failed (NT_STATUS_BAD_NETWORK_NAME)

when printing from my Mac to a printer hanging off a Ubuntu machine.

Having recently upgraded Ubuntu that was probably the culprit.

This link has more details http://ubuntuforums.org/showthread.php?t=1578421.

The short answer was to restart the following on the Ubuntu machine.

$sudo service cups restart
$sudo service smbd restart

Bookmark and Share
Posted in Uncategorized | Tagged , , , , , | 5 Comments