Getting Started with SUSHI:
For Developers of SUSHI Client Applications

In basic terms, SUSHI (ANSI/NISO Z39.93-2007, The Standardized Usage Statistics Harvesting Protocol) provides instructions to automate the collection of usage statistics reports that librarians would otherwise manually download from a vendor website or receive via e-mail. SUSHI is a relatively simple request-response protocol that calls for a client-server implementation. Typically, the usage statistics management software used by librarians will act as a client that sends a request to a web service implemented by a publisher or content provider. If you are a developer working for a library or an organization that is developing software to harvest and/or processes usage statistics, you will be developing the client application that sends requests to a the content provider’s SUSHI web service.

You will find lots of helpful information on the SUSHI website (www.niso.org/workrooms/sushi) that will make your job much easier -- THE MOST IMPORTANT OF WHICH IS THE LINK TO THE MISO CLIENT SOFTWARE. The MISO client is a working SUSHI client that is available through the Google Code site (http://code.google.com/p/sushicounterclient/). It includes an executable, complete configuration information and source code that can be used under the terms the New BSD License.

Another invaluable resource is the SUSHI Server Registry, which can be linked to from the NISO site or accessed directly at http://sites.google.com/site/sushiserverregistry/. This registry lists content providers offering SUSHI and provides details about what you need to do to use their SUSHI service.

Finally, remember that as you embark on this project you are not alone. The SUSHI Developer List (www.niso.org/lists/sushidevelopers/) provides you with access to a number of experts who can answer questions and provide guidance. And don’t forget the FAQs at www.niso.org/workrooms/sushi/faq/provider.

Here is a quick checklist of things you need to consider when developing your client application. You will notice that there are a number of things to take into consideration before you begin developing your client. 

Task/Consideration

Suggestions and Resources

Become familiar with the SUSHI protocol, SUSHI clients and the needs of SUSHI servers.

You’ve found this Getting Started Guide, so you already know there is lots of good background information on the NISO SUSHI site (http://www.niso.org/workrooms/sushi). This is a good place to start.

Next, download the MISO client and become familiar with its operation. The configuration file includes what is necessary to pull a sample report from Project Euclid.

Finally, review the SUSHI Server Registry and become familiar with the type of data that vendors require.

Become familiar with the information that goes into a SUSHI “Request”

A SUSHI Request does not have very many elements, but you do need to know what data goes where.  Following is a quick list and where you would find out:

Requestor ID:  provided by the content provider (see SUSHI Server Registry).

Customer ID: the data for the CustomerReference/ID element is provided by the content provider (see SUSHI Server Registry).

Report name: there is a standard set of names for the ReportDefinition/Name element. These are listed in the Reports Registry on the NISO SUSHI website.

Begin and End dates: the dates for the Begin and End elements (under ReportDefinition/Filters/UsageDate) must be in the format of “yyyy-mm-dd”. The End has to be greater than the Begin. Usage is pulled by the month so specify the first day of the Begin month and the last day of the End year.

Determine who will be using your application

Are you developing an application that will only be used by one institution; or, are you developing a service which needs to harvest and organize usage for many institutions?

The answer to this question will determine how robust a system you will need to develop. If you are developing a system for a single institution, you may be fine taking the approach of the MISO client, a command-line application that uses a simple text file to hold the configuration details for the servers to harvest data from. Whereas a solution that will manage usage for multiple institutions will need to consider databases to store each institutions details, user interfaces for managing those details and downloading usage, security, and more.

Determine if the application can be a stand-alone application or if it is integrated into another system.

With a stand-alone SUSHI client, you can harvest files and deposit them in a directory on a file system then allow some other application to ingest and process them. For example, some librarians are happy to use Excel to work with the files – they want SUSHI to just simplify getting those files.

If the SUSHI client is intended to be integrated in some larger usage consolidation/analysis application, then you will need to consider how these applications will work together and what kind of scheduling mechanism you will put in place.

Determine which reports you need to support.

The COUNTER Code of Practice (www.projectcounter.org/code_practice.html) covers a wide range of reports (not all vendors support all reports). Check with those who will be using your application to determine which reports they really need. It might be that Journal Report 1 is all that is needed.

If you are building a stand-alone SUSHI client this is less of an issue, however, if your SUSHI client is integrated into another application, you need to consider how that application will handle each of the reports.

Basically this analysis is about controlling the scope of your development.

Do you need reports automatically scheduled?

If you are developing a stand-alone client, it might be sufficient to just allow this application to be run on-demand. If, on the other hand, you are developing an integrated solution, you may want to plan for functionality that allows the harvesting to be scheduled. If you are planning on automatic scheduling, keep in mind that content providers may not all have their COUNTER data available on the same day of the month; therefore, you may want to account for that in the configuration data that you store.

Handling large reports

COUNTER reports can be big – VERY BIG. So big, in fact, that some common methods of handling XML objects may not work. Developers are urged to consider using something like the SAX Parser when processing the COUNTER reports – these parsers allow the report data to be “streamed” into the application rather than loaded completely into memory.

Handling errors

The SUSHI Response may be returned with an Error message. In some cases the error may indicate a problem with the request (something you will need to review and correct); other times it might indicate a problem at the Server side.

The Server may be informing you that it is busy or that the SUSHI reports are not ready yet. In these cases it is okay to retry the request; HOWEVER, it is important that you wait a reasonable amount of time. For example, if the Server reports that it is busy, this might indicate it is handling a lot of traffic or possibly that retrieving the data may take a while. It is recommended that you wait one HOUR before the retry in these cases. If you get another “Server Busy”, you may want to double this time before trying again, and so on.

If the Server is indicating that usage is not available at this time, you may want to wait 1 DAY before retrying.

As we mentioned in the previous consideration, COUNTER reports can be very big and may take time to process. Adding a level of forgiveness to the retries (e.g., waiting between retries), will provide a much better experience for your customer.

Develop the client

There are a number of tools on the NISO Tools and other Aids web page (http://www.niso.org/workrooms/sushi/tools). As mentioned above, the MISO client is a great place to start.

For .NET developers, EBSCO has provided an open source software developers toolkit that provides an excellent framework for doing your work and includes a basic client as well. Download the SDK from http://code.google.com/p/ebscosushisdk/  and start by reading the documentation in the zip file.

Test the results

EBSCO Information Services has included a free  Software Development Kit that includes a sample Server application.  You may want to start with this for experimenting with your client.

Another option is the Project Euclid server, which provides access to a test report.

Have others help with the testing

If you are developing the client for a particular institution, or are partnering with one or more institutions in the development of your client, you may want to contact some of the content providers listed in the SUSHI Server Registry to get the credentials and obtain access to their SUSHI Server so that you can test with live customer data.