Skip to content

Lumen API Documentation

Jump to bottom
Peter Hankiewicz edited this page Oct 28, 2024 · 46 revisions

The Lumen API allows 3rd parties to search the database and, if the 3rd party has an submitter authentication token, submit new notices to Lumen Database.

Searches are disabled unless you have an authentication token. Please see Authentication for information about searching with an authentication token.

The sample code below is provided for your convenience but will need to be modified for your use case. Unfortunately Lumen is unable to offer programming assistance.

  1. Authentication
  2. Request a Notice
  3. Request a List of Topics
  4. Search Notices via Fulltext
  5. Search for Entities via Fulltext
  6. Create Notice
  7. Submit Data
  8. Example Requests
  9. Example Responses
  10. Notice Type Mapping
  11. Scripting
  12. Troubleshooting

Authentication

  1. Examples: Getting Data From Lumen
  2. Examples: Sending Data to Lumen
  3. Troubleshooting
  4. Getting an API Token

Requests for notice data or searches through the API are not permitted without an Authentication Token. Submissions of data or notice creation through the API must be authenticated by including an Authentication Token.

With an authentication token, data requests are still throttled, but at a much higher limit (approximately one request per second). This limit was chosen to protect server resources while rarely or never interfering with most researchers' uses. However, if you are making a large number of automated requests, please sleep for at least one second between requests to avoid hitting the cap.

The Authentication Token may be included as a URL Parameter, an HTTP Request Header, or (when submitting data) as form data or JSON data.

Examples: Getting Data from Lumen

  • URL Parameter: authentication_token 
curl -kv -H "User-Agent: SomeUserAgentHere" https://lumendatabase.org/notices/1.json?authentication_token=abc123

curl -kv -H "User-Agent: SomeUserAgentHere" https://lumendatabase.org/notices/search.json?authentication_token=abc123

curl -kv -H "User-Agent: SomeUserAgentHere" "https://lumendatabase.org/notices/search.json?sender_name=James+Tiberius+Kirk&authentication_token=abc123"
  • HTTP Request Header: X-Authentication-Token 
curl -kv -H "User-Agent: SomeUserAgentHere" -H "X-Authentication-Token: abc123" https://lumendatabase.org/notices/1.json

curl -kv -H "User-Agent: SomeUserAgentHere" -H "X-Authentication-Token: abc123" https://lumendatabase.org/notices/search.json

Examples: Sending Data to Lumen

  • URL Parameter: authentication_token 
curl -H "User-Agent: SomeUserAgentHere" \
     -H "Accept: application/json" \
     -H "Content-type: application/json" \
     -d '{ "notice": {...} }' \
     'https://lumendatabase.org/notices?authentication_token=abc123'
  • Request Header: X-Authentication-Token 
curl -kv \
     -H "User-Agent: SomeUserAgentHere" \
     -H "X-Authentication-Token: abc123" \
     -F "notice[type]=Defamation" \
     https://lumendatabase.org/notices

curl -H "User-Agent: SomeUserAgentHere" \
     -H "Accept: application/json" \
     -H "Content-type: application/json" \
     -H "X-Authentication-Token: abc123" \
     -d '{ "notice": {...} }' \
     https://lumendatabase.org/notices

Troubleshooting

Did you get an HTTP 429 (Too Many Requests) response and a message saying you were browsing too fast?

Your API token is either missing or incorrectly applied. Check your request against the examples above.

Did you get a 'command not found' error using curl on the command line?

You may have to put the URL in quotes if it contains special characters.

Did you get an HTTP 401 (Unauthorized) response?

Your API token is either missing or incorrectly applied. Check your request against the examples above.

Getting an API token

Email [email protected] and describe your use case. Lumen API tokens are intended for research use only; please read the API Terms of Use before requesting a token.

Request a Notice

  1. Example request
  2. Successful Responses
  3. Example Successful Response
  4. Unsuccessful Responses

Method: GET

Endpoint: https://lumendatabase.org/notices/<notice id>.json

Example Request

curl -H "User-Agent: SomeUserAgentHere" https://lumendatabase.org/notices/1.json

⚠️ Note: a custom user agent is required. Default user agents such as "curl" are blocked by our servers.

You may have to put the URL in quotes if it contains special characters.

Successful Responses

Return a JSON-encoded representation of selected notice attributes. Notice Types will have mapped attributes applied, and be under a root key articulating their type.

Example Successful Response

{
  "dmca":{
    "id":1,
    "title":"Lion King on YouTube",
    "body":null,
    "date_sent":"2013-06-04T19:23:12Z",
    "date_received":"2013-06-05T20:31:44Z",
    "topics":[
      "Anticircumvention (DMCA)",
      "Bookmarks",
      "Lumen"
    ],
    "tags": [
      "tag_1",
      "tag_2"
    ],
    "jurisdictions": [
      "US",
      "CA"
    ],
    "action_taken": "Partial",
    "sender_name": "Joe Lawyer",
    "recipient_name": "Google, Inc.",
    "works": [
      {
        "description": "Lion King Video",
        "copyrighted_urls": [
          { "url": "http://www.example.com/lion_king.mp4" },
          { "url": "http://www.example.com/lion_king.mov" }
        ],
        "infringing_urls": [
          { "url": "http://www.example.com/infringing1" },
          { "url": "http://www.example.com/infringing2" },
          { "url": "http://www.example.com/infringing3" }
        ]
      }
    ]
  }
}

In cases where infringing or copyrighted URLs were not submitted to Lumen, the associated field will say [{ "url": "No URL submitted" }].

Unsuccessful Responses

Return a 404 HTTP status header.

Request a list of Topics

  1. Successful Responses
  2. Example Successful Response

Method: GET

Endpoint: https://lumendatabase.org/topics.json

curl -H "User-Agent: SomeUserAgentHere" https://lumendatabase.org/topics.json

Successful Responses

Return a JSON-encoded array of topics, including the following attributes:

Field Type Description
id Integer The unique ID used for the topic_ids array during notice creation.
name String The topic name
parent_id Integer The parent topic_id of this topic, or "null" if this is a root topic.

Example Successful Response

{
  "topics": [
    {
      "id": 1,
      "name": "Lumen",
      "parent_id": null
    },
    {
      "id": 2,
      "name": "Copyright",
      "parent_id": null
    },
    {
      "id": 3,
      "name": "Anticircumvention (DMCA)",
      "parent_id": 2
    }
  ]
}

Search notices via fulltext

  1. *-require-all Parameters
  2. Facets
  3. Troubleshooting very large searches
  4. Successful Responses
  5. Example Successful Response
  6. Example Unsuccessful Response

The Lumen website allows for paginatable full-text searches of notices and relevant metadata. Results are sorted with the most relevant at the top. Notice search results contain the same data as an individually-requested Notice, with an additional "score" field. "score" articulates how "relevant" this result is to the query term. Higher numbers are more relevant.

With an API token, results can be extensive; consider requesting gzipped content (if using curl, add -H "Accept-Encoding: gzip").

Lumen is unable to return past the 10,000th result due to limitations in Elasticsearch. This means that querying deeper than page=1000 (with the default per_page of 10) will fail.

Terms are joined with an 'OR' by default.

Method: GET

Endpoint: https://Lumendatabase.org/notices/search?term=url_escaped_query&sender_name=Joe%20Smith&page=1

Parameters

Parameters Description
term The full-text query term. To perform exact searches, enclose your search term in double quotes (""). This ensures that the search engine looks for the exact phrase as it is written. Searching for "full-text query" will return results containing precisely this phrase.
term-require-all If present, all words in the term query are required for a notice to be considered a match.
title Search in the title field
title-require-all If present, all words in the title query are required for a notice to be considered a match.
topics Search within a notice's topics
topics-require-all If present, all words in the topics query are required for a notice to be considered a match.
tags Search within a notice's tags
tags-require-all If present, all words in the tags query are required for a notice to be considered a match.
jurisdictions Search within a notice's jurisdictions
jurisdictions-require-all If present, all words in the jurisdictions query are required for a notice to be considered a match.
sender_name Search in the sender's name
sender_name-require-all If present, all words in the sender_name query are required for a notice to be considered a match.
principal_name Search in the principal's name
principal_name-require-all If present, all words in the principal_name query are required for a notice to be considered a match.
recipient_name Search in the recipient's name
recipient_name-require-all If present, all words in the recipient_name query are required for a notice to be considered a match.
works Search within a work's description
works-require-all If present, all words in the works query are required for a notice to be considered a match.
action_taken Search based on the action taken on a notice.
entities_country_codes Search within country codes of all notice's entities
topic_facet Filter on topics facet
sender_name_facet Filter on sender_name facet
principal_name_facet Filter on principal_name facet
recipient_name_facet Filter on recipient_name facet
tag_list_facet Filter on a tag
country_code_facet Filter on the submitter's country code
language_facet Filter on the notice language code
action_taken_facet Filter on the action_taken facet
date_received_facet The date range (in milliseconds since the unix epoch) separated by "..", e.g. 1546344000000..1546603200000, see Epoch format
date_submitted The date range (in milliseconds since the unix epoch) separated by "..", e.g. 1546344000000..1546603200000, see Epoch format
page The page you're requesting - defaults to the first page of results.
per_page The number of results per page. Defaults to 10.
sort_by One of date_received asc, date_received desc, relevancy asc, or relevancy desc. Defaults to relevancy asc.

*-require-all Parameters

Let's say we're searching for notices with "George Jetson" in the title. By default, the search engine will search for notices with "George OR Jetson".

If you include 'title-require-all=yes' in your query, then the search engine will search for notices with "George AND Jetson" in the title, narrowing down your results considerably.

The various *-require-all parameters need a non-null value to be enabled - "true" or "yes" are both acceptable.

Facets

See below for more information about available facets. You can get an idea of how facets are formatted by submitting facet-less fulltext queries and then inspecting the Facets metadata returned by the search.

Troubleshooting very large searches

Searches with a very large number of results (e.g. "DMCA") may fail. If this happens, you will see the URL for your search in the address bar, but a "REQUEST REFUSED (500)" error in the page.

Fix this by narrowing your search. For instance, add a date facet to the end of the URL, such as &date_received_facet=1586404800000.0..1602216000000.0. This example yields results from April 9, 2020, through October 9, 2020. See Epoch format below for how to generate good numbers for this facet, or simply append this example to your URL and you should be taken back to a search results page which you can then tweak as you wish.

Successful Responses

Return a JSON-encoded hash including an array of notices and metadata about the search results.

Field Type Description
notices Array An array of Notices encoded as JSON data structures.
meta Hash Search Metadata about the results of the search. See Search Metadata.

Search Metadata

Field Type Description
query Query The search query meta information. See Query.
facets Facet How the result set "falls" into metadata facets. See Facets
current_page Integer The page number of the current set of results.
next_page Integer The page number of the next set of results, or null if this is the last page.
offset Integer How many total results in the result set before the current list of results.
per_page Integer Number of results per page
previous_page Integer The page number of the previous set of results, or null if this is the first page.
total_entries Integer The total number of results for the search query.
total_pages Integer The total number of pages in this result set.

Query

Field Type Description
term String The full-text search term
sender_name_facet String The sender_name value if this facet was submitted in this search.
recipient_name_facet String The recipient_name value if this facet was submitted in this search.
topic_facet String The topic value if this facet was submitted in this search.
date_received_facet String The date range used (in milliseconds since the unix epoch) separated by "..", e.g. 1546344000000..1546603200000, see Epoch format
tag_list_facet String The tag_list value if this facet was submitted in this search.
country_code_facet String The country_code value if this facet was submitted in this search.
language_facet String The language value if this facet was submitted in this search.

Facet

Facets aggregate documents along specific metadata. See the elasticsearch documentation for more information on facets.

Currently we're using value and range facets.

Facet Facet Type Description
sender_name_facet Terms The top 10 sender_names in this result set.
recipient_name_facet Terms The top 10 recipient_names in this result set.
topic_facet Terms The top 10 topics in this result set.
tag_list_facet Terms The top 10 tags in this result set.
country_code_facet Terms The top 10 country_codes in this result set.
language_facet Terms The top 10 languages in this result set.
date_received Range The available date range facets.

Terms Facet

Only the most relevant metadata is described below. See the elasticsearch documentation for more information. We only return the 10 most populous facets.

Field Type Description
total Integer The total number of results that can be faceted on this term and query.
other Integer The number of results not included in the facets returned.
terms Array An array including the term and count of items that are in that term facet.

Range Facet

Field Type Description
from Integer The start of this facet, in unix epoch milliseconds, e.g. 1546344000000, see Epoch format
to Integer The end of this facet, in unix epoch milliseconds, e.g. 1546603200000, see Epoch format
from_str String A textual representation of the start of this facet.
to_str String A textual representation of the end of this facet.
count Integer The number of documents that fall in this facet.

Example Successful Response

Note: Some facet information has been omitted to keep examples brief.

curl -H "User-Agent: SomeUserAgentHere" -H "Accept: application/json" -H "Content-type: application/json" 'https://lumendatabase.org/notices/search?term=star'

{
  "notices": [
    {
      "id": "491",
      "type": "DMCA",
      "title": "Star Wars",
      "body": null,
      "date_received": null,
      "language": "en",
      "topics": [
        "Trademark",
        "DMCA Notices",
        "John Doe Anonymity"
      ],
      "sender_name": "Joe Lawyer",
      "recipient_name": "Google, Inc.",
      "tags": [
        "tag_1",
        "tag_2"
      ],
      "jurisdictions": [
        "US",
        "CA"
      ],
      "works": [
        {
          "description": "Lion King Video",
          "copyrighted_urls": [
            { "url": "http://www.example.com/lion_king.mp4" },
            { "url": "http://www.example.com/lion_king.mov" }
          ],
          "infringing_urls": [
            { "url": "http://www.example.com/infringing1" },
            { "url": "http://www.example.com/infringing2" },
            { "url": "http://www.example.com/infringing3" }
          ]
        }
      ],
      "score": 0.64636606
    }
  ],
  "meta": {
    "query": {
      "term": "star"
    },
    "facets": {
      "sender_name_facet": {
        "_type": "terms",
        "missing": 0,
        "total": 7,
        "other": 0,
        "terms": [
          {
            "term": "Steve Simpson",
            "count": 5
          },
          {
            "term": "Mike Itten",
            "count": 2
          }
        ]
      },
      "recipient_name_facet": {
        "_type": "terms",
        "missing": 0,
        "total": 7,
        "other": 0,
        "terms": [
          {
            "term": "Google",
            "count": 5
          },
          {
            "term": "Twitter",
            "count": 2
          }
        ]
      },
    },
    "current_page": 1,
    "next_page": null,
    "offset": 0,
    "per_page": 10,
    "previous_page": null,
    "total_entries": 1,
    "total_pages": 1
  }
}

Example Unsuccessful Response

{
  "notices": [ ],
  "meta": {
    "query": {
      "term": "nonexistent"
    },
    "current_page": 1,
    "next_page": null,
    "offset": 0,
    "per_page": 10,
    "previous_page": null,
    "total_entries": 0,
    "total_pages": 0
  }
}

Search for Entities via fulltext

  1. Successful Responses
  2. Example Successful Response
  3. Example Unsuccessful Response

The Lumen database allows for paginatable full-text searches of Entities, useful to select existing entities during notice creation. This method is only available via the API, not the web site. For information about pagination metadata, please see Search notices via fulltext.

Method: GET

Endpoint: https://lumendatabase.org/entities/search.json?term=url_escaped_query&page=1

Parameters

Parameters Description
term The full-text query term
page The page you're requesting - defaults to the first page of results.
per_page The number of results per page. Defaults to 10.

Successful Responses

Return a JSON-encoded hash including an array of entities and metadata about the search results.

Field Type Description
entities Array An array of Entities encoded as JSON data structures.
meta Hash Search Metadata about the results of the search. See Search Metadata.

Entity

Field Type Description
id string The unique ID
parent_id string The parent ID or "null" if this is a "root" entity.
name string Full name
address_line_1 string
address_line_2 string
state string
country_code string Ideally, an ISO country code.
phone string
email string
url string
city string

Example Successful Response

curl -H "User-Agent: SomeUserAgentHere" -H "Accept: application/json" -H "Content-type: application/json" 'https://lumendatabase.org/entities/search?term=joe&page=1'

{
  "entities": [
    {
      "id": "6",
      "parent_id": null,
      "name": "Steve Simpson",
      "address_line_1": "23 My St.",
      "address_line_2": null,
      "state": "CA",
      "country_code": "US",
      "phone": null,
      "email": null,
      "url": null,
      "city": "Scranton"
    }
  ],
  "meta": {
    "query": {
      "term": "simpson"
    },
    "facets": null,
    "current_page": 1,
    "next_page": null,
    "offset": 0,
    "per_page": 10,
    "previous_page": null,
    "total_entries": 1,
    "total_pages": 1
  }
}

Example Unsuccessful Response

{
  "entities": [ ],
  "meta": {
    "query": {
      "term": "nonexistent"
    },
    "facets": null,
    "current_page": 1,
    "next_page": null,
    "offset": 0,
    "per_page": 10,
    "previous_page": null,
    "total_entries": 0,
    "total_pages": 0
  }
}

Create notice

Submits a new Notice to the Lumen system.

There are several different "Notice Types"; these are essentially subclasses of the abstract Notice class. "Notice Types" allow us to track notice-specific attributes and create serialized representations with logical attribute names. Please see Notice Type Mapping, which defines what attributes are remapped for each Notice Type.

An authentication token is required to submit via the API. Please ask Lumen staff for your authentication token. Use of the the authentication token is described in the Authentication section.

Method: POST

Endpoint: https://lumendatabase.org/notices

Request

Field Type Description
notice Notice Required
authentication_token string See Authentication

Notice

Note: The notice types must be sent exactly as written here, i.e., CamelCase with no spaces except DMCA which is all-caps.

Field Type Description
title string Required
type string Required: one of 'Counternotice', 'CourtOrder', 'DataProtection', 'Defamation', 'DMCA', 'LawEnforcementRequest', 'Other', 'PrivateInformation', 'GovernmentRequest', or 'Trademark'
subject string Optional. A short description of the notice and its contents. E.g. "DMCA Notice Regarding Photographs" or "Court Order From Paris Court Re: Hate Speech"
body string Optional. Use this field to submit any additional text provided by the complainant that does not belong properly in any other notice field. No sensitive information or PII should be included here. E.g., Phone numbers, street addresses, allegedly defamatory language, etc.
date_sent string Any parseable time value, e.g. "2013-05-21", "2013-05-21 10:01:01 -04:00"
language string A two character language code. See "Language" below for the list.
date_received string Any parseable time value, e.g. "2013-05-21", "2013-05-21 10:01:01 -04:00"
source string How did you receive this notice - mail? online form?
topic_ids [int] An array of integers representing topic_ids - See Request a list of Topics.
tag_list string Comma separated tags, spaces are OK. Automatically lowercased.
regulation_list string Comma separated regulations / laws relevant to this notice, spaces are OK. Automatically lowercased. Only available for CourtOrder, and LawEnforcementRequest notice types.
jurisidiction_list string Comma separated list of jurisdictions, spaces are OK.
action_taken string One of 'Yes', 'No', or 'Partial'.
url_count string This field is used to indicate the total number of URLs contained in a particular Data Protection request. I.e. If the requester asked for the removal of 10 URLs, this should be set to "10". If no value is set, this will display as "unspecified".
request_type string One of 'Agency', 'Civil Subpoena', 'Email', 'Records Preservation', 'Subpoena', 'Warrant'. Valid only for LawEnforcementRequest notices.
mark_registration_number string A mark registration number. Valid only for Trademark notices.
works_attributes [Work] A list of Works
entity_notice_roles_attributes [EntityNoticeRole] A list of EntityNoticeRoles
file_uploads_attributes [FileUpload] A list of FileUploads
case_id_number int Optional, a court case number, specific to the CourtOrder type

Work

Field Type Description
copyrighted_urls_attributes [CopyrightedUrl] List of URLs that represent the original work.
kind string Required. Book, movie, video, etc.
description string Description of the work, which may include the copyright holder information
infringing_urls_attributes [InfringingUrl] List of URLs which infringe on the work

CopyrightedURL

Field Type Description
url string A URL that represents the original work. 8 kilobyte limit.

InfringingUrl

Field Type Description
url string A URL that infringes upon the Work. 8 kilobyte limit.

EntityNoticeRole

Field Type Description
name string The name of the role, one of "principal", "agent", "recipient", "sender", "target", "issuing_court", "plaintiff", "defendant" or "submitter". "recipient" and "sender" are displayed for all Notice Types on the Lumen website after automatic redaction.
entity_id integer (Optional) The ID of an existing entity. You should not specify the entity_attributes values below if you include entity_id here.
entity_attributes Entity A new entity that has this role on this notice. If you specify entity_attributes, you should not specify an entity_id

Note: If not explicitly provided, recipient entities will be assigned based on the user performing the submission (as identified by the authentication token used).

The submitter entity will always be assigned based on the user performing the submission (as identified by the authentication token used).

Entity

Field Type Description
name string Required
kind string one of "organization" or "individual"
address_line_1 string
address_line_2 string
city string
state string
zip string
country_code string A two-digit ISO country code
phone string
email string
url string

FileUpload

File uploading can only be used when submitting as form data. See the Submit Data section.

Field Type Description
kind string One of "original" or "supporting"
file string Path to a file on the client's system.

Language

The currently supported two-digit language codes are based on what Google Translate supports, and are currently:

iso Country iso Country
af Afrikaans ar Arabic
be Belarusian bg Bulgarian
ca Catalan; Valencian cs Czech
cy Welsh da Danish
de German el Greek, Modern
en English eo Esperanto
es Spanish; Castilian et Estonian
fa Persian fi Finnish
fr French ga Irish
gl Galician hi Hindi
hr Croatian ht Haitian; Haitian Creole
hu Hungarian id Indonesian
is Icelandic it Italian
iw Hebrew ja Japanese
ko Korean lt Lithuanian
lv Latvian mk Macedonian
ml Malayalam ms Malay
mt Maltese nl Dutch
no Norwegian pl Polish
pt Portuguese ro Romanian
ru Russian si Sinhala
sk Slovak sl Slovene
sq Albanian sr Serbian
sv Swedish sw Swahili
th Thai tl Tagalog
tr Turkish uk Ukrainian
vi Vietnamese yi Yiddish
yo Yoruba zh Chinese

Submit Data

  1. As Form Data
  2. As JSON Data

When creating a notice through the API, there are two ways to submit notice data: as form data or as JSON data. Each has advantages and disadvantages described below.

Data submission must include an authentication token (see Authentication).

As Form Data

Form data is useful for attaching files to the notice as the files can be included by name and location on a local drive.

This is the suggested method as it is more efficient at attaching files, which are transmitted as binary data.

Sample using curl

In curl, use the @ symbol to specify a local file.

curl -kv \
     -H "User-Agent: SomeUserAgentHere" \
     -H "X-Authentication-Token: abc123" \
     -F "notice[type]=Defamation" \
     -F "notice[title]=Defamation report via form data" \
     -F "notice[file_uploads_attributes][0][kind]=original" \
     -F "notice[file_uploads_attributes][0][file]=@Defamation complaint.docx" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://www.bad.com/people/296" \
     https://lumendatabase.org/notices

As JSON Data

JSON data is better if you do not have to submit file attachments as the syntax is simpler.

curl -H "User-Agent: SomeUserAgentHere" \
     -H "Accept: application/json" \
     -H "X-Authentication-Token: abc123" \
     -H "Content-type: application/json" \
     -d '{ "notice": { "type": "Defamation", "title": "Defamation report via JSON data" } }' \
     https://lumendatabase.org/notices

Example Requests

  1. Example Request Using Entity Attributes
    1. As Form Data
    2. As JSON Data
  2. Example Request Using Entity ID
    1. As Form Data
    2. As JSON Data

Example Request Using Entity Attributes

As Form Data

curl -kv \
     -H "User-Agent: SomeUserAgentHere"
     -H "X-Authentication-Token: abc123" \
     -F "notice[type]=DMCA" \
     -F "notice[title]=DMCA (Copyright) Complaint to Google" \
     -F "notice[subject]=Infringement Notfication via Blogger Complaint" \
     -F "notice[date_sent]=2014-07-04" \
     -F "notice[date_received]=2014-07-04" \
     -F "notice[source]=Online form" \
     -F "notice[body]=Victim of Infringement requests removal of content on Blogger." \
     -F "notice[topic_ids][]=19", \
     -F "notice[topic_ids][]=27", \
     -F "notice[tag_list]=movies, disney, youtube", \
     -F "notice[jurisdiction_list]=US, CA", \
     -F "notice[action_taken]=Partial", \
     -F "notice[file_uploads_attributes][0][kind]=original" \
     -F "notice[file_uploads_attributes][0][file]=@original_file.txt" \
     -F "notice[file_uploads_attributes][1][kind]=supporting" \
     -F "notice[file_uploads_attributes][1][file]=@supporting_file.txt" \
     -F "notice[works_attributes][0][description]=Lion King Video" \
     -F "notice[works_attributes][0][copyrighted_urls_attributes][0][url]=http://disney.com/lion_king.mp4" \
     -F "notice[works_attributes][0][copyrighted_urls_attributes][0][url]=http://disney.com/lion_king.mov" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_1" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_2" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_3" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_4" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_5" \
     -F "notice[entity_notice_roles_attributes][0][name]=recipient" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][name]=Google" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][kind]=organization" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][address_line_1]=1600 Amphitheatre Parkway" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][city]=Mountain View" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][state]=CA" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][zip]=94043" \
     -F "notice[entity_notice_roles_attributes][0][entity_attributes][country_code]=US" \
     -F "notice[entity_notice_roles_attributes][1][name]=sender" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][name]=Joe Lawyer" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][kind]=individual" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][address_line_1]=1234 Anystreet St." \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][city]=Anytown" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][state]=CA" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][zip]=94044" \
     -F "notice[entity_notice_roles_attributes][1][entity_attributes][country_code]=US" \
     https://lumendatabase.org/notices

As JSON Data

curl -v \
  -H "User-Agent: SomeUserAgentHere" \
  -H "Accept: application/json" \
  -H "Content-type: application/json" \
  'https://lumendatabase.org/notices/' \
  -d '{
  "notice": {
    "title": "DMCA (Copyright) Complaint to Google",
    "type": "DMCA",
    "subject": "Infringement Notfication via Blogger Complaint",
    "date_sent": "2013-05-22",
    "date_received": "2013-05-23",
    "source": "Online form",
    "topic_ids": [19, 27],
    "tag_list": "movies, disney, youtube",
    "jurisdiction_list": "US, CA",
    "action_taken": "Partial",
    "works_attributes": [
      {
        "description": "Lion King Video",
        "copyrighted_urls_attributes": [
          { "url": "http://disney.com/lion_king.mp4" },
          { "url": "http://disney.com/lion_king.mov" }
        ],
        "infringing_urls_attributes": [
          { "url": "http://youtube.com/bad_url_1" },
          { "url": "http://youtube.com/bad_url_2" },
          { "url": "http://youtube.com/bad_url_3" },
          { "url": "http://youtube.com/bad_url_4" },
          { "url": "http://youtube.com/bad_url_5" }
        ]
      }
    ],
    "entity_notice_roles_attributes": [
      {
        "name": "recipient",
        "entity_attributes": {
          "name": "Google",
          "kind": "organization",
          "address_line_1": "1600 Amphitheatre Parkway",
          "city": "Mountain View",
          "state": "CA", 
          "zip": "94043",
          "country_code": "US"
        }
      },
      {
        "name": "sender",
        "entity_attributes": {
          "name": "Joe Lawyer",
          "kind": "individual",
          "address_line_1": "1234 Anystreet St.",
          "city": "Anytown",
          "state": "CA",
          "zip": "94044",
          "country_code": "US"
        }
      }
    ]
  }
}'

Example Request Using Entity ID

As Form Data

curl -kv \
     -H "User-Agent: SomeUserAgentHere" \
     -H "X-Authentication-Token: abc123" \
     -F "notice[type]=DMCA" \
     -F "notice[title]=DMCA (Copyright) Complaint to Google" \
     -F "notice[subject]=Infringement Notfication via Blogger Complaint" \
     -F "notice[date_sent]=2014-07-04" \
     -F "notice[date_received]=2014-07-04" \
     -F "notice[source]=Online form" \
     -F "notice[body]=Victim of Infringement requests removal of content on Blogger." \
     -F "notice[topic_ids][]=19", \
     -F "notice[topic_ids][]=27", \
     -F "notice[tag_list]=movies, disney, youtube", \
     -F "notice[jurisdiction_list]=US, CA", \
     -F "notice[action_taken]=Partial", \
     -F "notice[file_uploads_attributes][0][kind]=original" \
     -F "notice[file_uploads_attributes][0][file]=@original_file.txt" \
     -F "notice[file_uploads_attributes][1][kind]=supporting" \
     -F "notice[file_uploads_attributes][1][file]=@supporting_file.txt" \
     -F "notice[works_attributes][0][description]=Lion King Video" \
     -F "notice[works_attributes][0][copyrighted_urls_attributes][0][url]=http://disney.com/lion_king.mp4" \
     -F "notice[works_attributes][0][copyrighted_urls_attributes][0][url]=http://disney.com/lion_king.mov" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_1" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_2" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_3" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_4" \
     -F "notice[works_attributes][0][infringing_urls_attributes][0][url]=http://youtube.com/bad_url_5" \
     -F "notice[entity_id]=1" \
     https://lumendatabase.org/notices

As JSON data

curl -v \
  -H "User-Agent: SomeUserAgentHere" \
  -H "Accept: application/json" \
  -H "Content-type: application/json" \
  -H "X-Authentication-Token: abc123" \
  'https://lumendatabase.org/notices/' \
  -d '{
  "notice": {
    "title": "DMCA (Copyright) Complaint to Google",
    "type": "DMCA",
    "subject": "Infringement Notfication via Blogger Complaint",
    "date_sent": "2013-05-22",
    "date_received": "2013-05-23",
    "source": "Online form",
    "topic_ids": [19, 27],
    "tag_list": "movies, disney, youtube",
    "jurisdiction_list": "US, CA",
    "action_taken": "Partial",
    "works_attributes": [
      {
        "description": "Lion King Video",
        "copyrighted_urls_attributes": [
          { "url": "http://disney.com/lion_king.mp4" },
          { "url": "http://disney.com/lion_king.mov" }
        ],
        "infringing_urls_attributes": [
          { "url": "http://youtube.com/bad_url_1" },
          { "url": "http://youtube.com/bad_url_2" },
          { "url": "http://youtube.com/bad_url_3" },
          { "url": "http://youtube.com/bad_url_4" },
          { "url": "http://youtube.com/bad_url_5" }
        ]
      }
    ],
    "entity_id": 1
  }
}'

Example Responses

  1. Successful Responses
  2. Unsuccessful Responses

Successful Responses

Successful responses will have HTTP status 201 (Created) and include a Location: HTTP header with the location of the created object.

Note: Unimportant headers have been removed.

HTTP/1.1 201 Created 
Location: http://localhost:3000/notices/4
Content-Type: application/json; charset=utf-8
Cache-Control: max-age=0, private, must-revalidate
Date: Wed, 05 Jun 2013 20:43:34 GMT
Content-Length: 1

Unsuccessful Responses

Most unsuccessful responses will have HTTP status 422 (Unprocessable Entity) and include a JSON response enumerating the validation failures.

If requests are too fast or too numerous, various other HTTP error codes may appear. If these are persistent, feel free to contact us and we will look into it. Be sure to include your IP address, user agent, and API key, if any.

{
  "works":["can't be blank"],
  "entity_notice_roles":["can't be blank"],
  "title":["can't be blank"]
}

Notice Type Mapping

  1. Counternotice
  2. CourtOrder
  3. DataProtection
  4. Defamation
  5. DMCA
  6. GovernmentRequest
  7. LawEnforcementRequest
  8. Other
  9. PrivateInformation
  10. Trademark

All notices are added to the system through the API using the same named parameters (see Create Notice). However, fields are "mapped" depending on the Notice Type when serialized as JSON through a search request or when requested individually.

Counternotice

Attributes remain unchanged from ingestion.

CourtOrder

Ingestion Field Name Mapped Field Name Removed?
body explanation
regulations laws_referenced
works.description works.subject_of_court_order
works.infringing_urls works.targetted_urls
works.copyrighted_urls Yes

DataProtection

Ingestion Field Name Mapped Field Name Removed?
body legal_complaint
works.description works.complaint
works.infringing_urls works.urls_mentioned_in_request
works.copyrighted_urls Yes

Defamation

Ingestion Field Name Mapped Field Name Removed?
body legal_complaint
works.infringing_urls works.defamatory_urls
works.copyrighted_urls Yes

DMCA

Attributes remain unchanged from ingestion.

GovernmentRequest

Ingestion Field Name Mapped Field Name Removed?
body explanation
works.description works.subject
works.infringing_urls works.urls_of_original_work
works.copyrighted_urls works.urls_mentioned_in_request

LawEnforcementRequest

Ingestion Field Name Mapped Field Name Removed?
body explanation
works.description works.subject_of_enforcement_request
works.infringing_urls works.urls_in_request
works.copyrighted_urls works.original_work_urls
regulation_list regulations

Other

Ingestion Field Name Mapped Field Name Removed?
body explanation
works.description works.complaint
works.infringing_urls works.problematic_urls
works.copyrighted_urls works.original_work_urls

PrivateInformation

Ingestion Field Name Mapped Field Name Removed?
body explanation
works.description works.complaint
works.infringing_urls works.urls_with_private_information
works.copyrighted_urls Yes

Trademark

Ingestion Field Name Mapped Field Name Removed?
mark_registration_number
works.description marks.description
works.infringing_urls marks.infringing_urls
works.copyrighted_urls Yes

Epoch format

In some of the facets we use the Unix epoch time format (https://en.wikipedia.org/wiki/Unix_time) in milliseconds.

An easy way to convert time from a human-readable format to epoch is using the https://www.epochconverter.com website. Use the converter to convert it to milliseconds.

Tuesday, 1 January 2019 12:00:00 = 1546344000000 milliseconds.

Scripting

Lumen Database does not offer packaged downloads of data but we also do not forbid researchers from writing scripts to perform searches or retrieve notice data. However, we do have a few suggestions that will help keep our system responsive and also help us if you run into any issues.

  1. Either request .json endpoints or include the "Accept: application/json" header.
  2. Consider adding an accept-encoding header to indicate you prefer compressed results (in curl, -H "Accept-Encoding: gzip").
  3. Make sure you are using your authentication token to avoid rate limiting.
Clone this wiki locally