Talkwalker API Overview
Talkwalker Search API Overview & Example
How it works
The Talkwalker Search API allows you to retrieve up to 500 sorted results for a given timeframe within the last 30 days. In addition, a histogram of the number of results can also be returned. You can sort the results by publication time, indexing time, engagement or other metrics. A single search query can support up to 50 operands. To create complex queries, operands may be combined using Boolean operators.
A few words about the results
Search results can be sorted by engagement, time or other metrics and be restricted to specific attribute value ranges (for example only return results published in a certain timerange). When no special filters are applied, a single search request will return results from all media types and all languages over the past 30 days sorted by engagement by default. You don’t need to execute one search request for each language and media type separately. To get a smaller set of results, you can either get only the highest ranked results or get a random sample set.
A brief example (Search)
The Talkwalker API search results endpoint (https://api.talkwalker.com/api/v1/search/results) is used to search on the Talkwalker API. (For testing purpose the access_token demo can be used. Setting the variable pretty=true will return formatted results)
curl -XGET 'https://api.talkwalker.com/api/v1/search/results?access_token=demo&q=cats&pretty=true'response (all responses are UTF-8):
{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v1/search/results?access_token=demo&q=cats&pretty=true",
"pagination" : {
"next" : "GET /api/v1/search/results?access_token=demo&q=cats&pretty=true&offset=10",
"total" : 298138
},
"result_content" : {
"data" : [ {
"data" : {
"url" : "http://annukcreations.blogspot.com/2014/12/sunny-rings.html",
"indexed" : 1417999367498,
"search_indexed" : 1417999504832,
"published" : 1417999319393,
"title" : "Color and Light Inspirations in Jewelry: SUNNY RINGS :)",
"content" : "Welcome to my colorful little island (...)",
"title_snippet" : "Color and Light Inspirations in Jewelry: SUNNY RINGS :)",
"root_url" : "http://annukcreations.blogspot.com/",
"domain_url" : "http://blogspot.com/",
"host_url" : "http://annukcreations.blogspot.com/",
"parent_url" : "http://annukcreations.blogspot.com/2014/12/sunny-rings.html",
"lang" : "en",
"porn_level" : 0,
"fluency_level" : 90,
"spam_level" : 20,
"sentiment" : 5,
"source_type" : [ "BLOG", "BLOG_OTHER" ],
"post_type" : [ "TEXT" ],
"tokens_title" : [ "and Light Inspirations", "Light Inspirations", "Light Inspirations", "SUNNY RINGS", "SUNNY RINGS", "and Light", "Inspirations", "Inspirations", "RINGS", "RINGS", "Light", "Light", "Jewelry", "Jewelry", "Color", "Color", "SUNNY", "SUNNY" ],
"tokens_content" : [ "Bead Hoarder Blog", "Bead Hoarder Blog"],
"tokens_mention" : [ "@yahoo" ],
"tags_internal" : [ "isQuestion" ],
"article_extended_attributes" : {
"num_comments" : 3
},
"source_extended_attributes" : {
"alexa_pageviews" : 0
},
"extra_article_attributes" : {
"world_data" : { }
},
"extra_author_attributes" : {
"world_data" : { },
"id" : "ex:annukcreations.blogspot.com-698904645",
"name" : "view my complete profile",
"gender" : "MALE"
},
"extra_source_attributes" : {
"world_data" : {
"continent" : "North America",
"country" : "United States",
"region" : "District of Columbia",
"city" : "Washington, D.C.",
"longitude" : -77.0094185808,
"latitude" : 38.8995493765,
"country_code" : "us"
},
"id" : "ex:annukcreations.blogspot.com",
"name" : "http://annukcreations.blogspot.com/"
},
"engagement" : 3,
"reach" : 0
}
}, {
"data" : {
"url" : "http://slshoeicidal.wordpress.com/2014/12/06/high-rez-snobbery-715-winter-trend-ice/",
... // truncatedmore on the Talkwalker Search Results API
Talkwalker Streaming API Overview & Example
How it works
The Talkwalker Streaming API delivers real-time data through a persistent connection to our servers. Configure your stream with a set of filtering rules, connect to the stream and new results will be delivered in real time, as soon as they are found by our crawlers. You will not need to do any polling to receive new data.
You setup and configure the Streaming API by defining rules (Boolean query, language, media types, etc.). The Streaming API then finds and collects all relevant data and adds it to your data stream, with individually highlighted snippets per matched rule. This feature allows you to gather data from many rules through a single stream while easily matching the results back to your predefined rules.
Each rule allows filtering by title, content, author, language, URL, country, media type, and more parameters, using the same syntax as in our Talkwalker Search interface. You can also apply a list of sources to be included or excluded from the stream, to give you even further possibilities to narrow down the results you will get. A single rule can support up to 50 operands. To create complex rules, operands may be combined using Boolean Operators.
The documents are streamed in the order they are found by our crawlers and added to Talkwalker (i.e. by search_indexed timestamp). Custom sorting is not possible with the Streaming API (however this can be done with the Search API). The documents are grouped in timeframes which contain all documents that were indexed between the given start and end time of the timeframe.
Each result (independent on how many rules match) will be counted as 1 credit.
A brief example (Streaming)
The Talkwalker API streaming endpoint (https://api.talkwalker.com/api/v3/stream) is used to stream results from Talkwalker.
Creating a Stream
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/s/teststream?access_token=demo'
-d '{ "rules" : [{ "rule_id": "rule-1", "query": "cats" }] }'
-H 'Content-Type: application/json; charset=UTF-8'{
"status_code" : "0",
"status_message" : "OK",
"request" : "PUT /api/v3/stream/s/teststream?access_token=demo",
"result_stream" : {
"data" : [{
"stream_id" : "teststream",
"rules" : [{
"rule_id" : "rule-1",
"query" : "cats"
}]
}]
}
}Streaming
curl -XGET 'https://api.talkwalker.com/api/v3/stream/s/teststream/results?access_token=demo'The response is a stream of chunks, chunks contain meta data (CT_CONTROL) on the Talkwalker stream or search results (CT_RESULT).
{
"chunk_type" : "CT_CONTROL",
"chunk_control" : {
"stream" : [ {
"id" : "teststream",
"status" : "active"
} ],
"connection_id" : "#piv2tmbz2yxu#"
}
}
{
"chunk_type": "CT_RESULT",
"chunk_result": {
"data" : {
"data" : {
"url" : "http://annukcreations.blogspot.com/2014/12/sunny-rings.html",
"indexed" : 1417999367498,
"search_indexed" : 1417999504832,
"published" : 1417999319393,
"title" : "Color and Light Inspirations in Jewelry: SUNNY RINGS :)",
"content" : "Welcome to my colorful little island (...)",
"title_snippet" : "Color and Light Inspirations in Jewelry: SUNNY RINGS :)",
"root_url" : "http://annukcreations.blogspot.com/",
"domain_url" : "http://blogspot.com/",
"host_url" : "http://annukcreations.blogspot.com/",
"parent_url" : "http://annukcreations.blogspot.com/2014/12/sunny-rings.html",
"lang" : "en",
... // truncatedmore on the [_talkwalker_streaming_api]
Talkwalker Search API
Talkwalker Search Results API
https://api.talkwalker.com/api/v1/search/resultsHow it works
The Talkwalker Search API allows you to retrieve up to 500 sorted results for a given timeframe within the last 30 days. In addition, a histogram of the number of results can also be returned. You can sort the results by publication time, indexing time, engagement or other metrics. A single search query can support up to 50 operands. To create complex queries, operands may be combined using Boolean operators.
A few words about the results
Search results can be sorted by engagement, time or other metrics and be restricted to specific attribute value ranges (for example only return results published in a certain timerange). When no special filters are applied, a single search request will return results from all media types and all languages over the past 30 days sorted by engagement by default. You don’t need to execute one search request for each language and media type separately. To get a smaller set of results, you can either get only the highest ranked results or get a random sample set.
Parameters
| parameter | description | required? | default value |
|---|---|---|---|
|
API access token |
required |
|
|
The query to search for |
required |
|
|
Number of results to skip (for paging) |
optional |
default: 0 |
|
Number of hits per page (for paging) |
optional |
default: 10 / maximum : 500 |
|
Criteria for sorting the results. |
optional |
default: engagement |
|
Sorting order (ascending or descending) |
optional |
default: desc |
|
Turns highlighting on or off |
optional |
default: 1 |
|
Formatted json for testing |
optional |
false |
More on the Talkwalker Query Syntax
Credits
1 credit per returned result, at least 10 credits per call (e.g. 100 results = 100 credits, 10 results = 10 credits and 0 results = 10 credits).
Examples
Get 100 results containing the words "cats" and "dogs" but not "birds"
Set the query cats AND dogs AND NOT birds with query=cats%20AND%20dogs%20AND%20NOT%20birds (note: in URLs spaces are replaced by %20) and set hits per page to 100 with hpp=100.
curl 'https://api.talkwalker.com/api/v1/search/results?access_token=demo&q=cats%20AND%20dogs%20AND%20NOT%20birds&hpp=100'More on the Talkwalker Query Syntax
Get results containing the word "cats" published in american blogs
curl 'https://api.talkwalker.com/api/v1/search/results?access_token=demo&q=cats%20AND%20sourcetype:"BLOG"%20AND%20sourcecountry:us'Get results containing the word "cats" sorted from new to old
To sort the results by date, set sort_by to published (to sort by the date of publication), to get the newest results first, set sort_order=desc.
curl 'https://api.talkwalker.com/api/v1/search/results?access_token=demo&q=cats&sort_by=published&sort_order=desc'All options for sort_by are :
reach, facebook_shares, facebook_likes, twitter_shares, twitter_retweets,
twitter_followers, youtube_likes, youtube_dislikes, youtube_views,
cluster_size, comment_count, published, search_indexed, trending_score
You may find additional information on the document fields , except for trending_score, as presented next.
What is the meaning of Trending Score?
The Trending Score evaluates the acceleration of the engagement of a specific story over time. Mentions which are rapidly getting more engagement are given a higher score (out of 10), enabling you to discover breaking stories on specific topics. When calculating the Trending Score our algorithms look at the speed at which the engagement is growing, and then assign each result with a relative score (out of 10) based upon the speed at which the engagement is growing. The scores are categorised in the following way:
-
0-3: the article is not trending (or not trending anymore)
-
4-6: the article is slightly trending
-
7-10: the article is really trending, right now
Get results containing the word "health" sorted by their Trending Score
curl 'https://api.talkwalker.com/api/v1/search/results?access_token=demo&q=health&sort_by=trending_score&sort_order=desc&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v1/search/results?access_token=demo&q=health&sort_by=trending_score&sort_order=desc&pretty=true",
"pagination" : {
"next" : "GET /api/v1/search/results?access_token=demo&q=health&sort_by=trending_score&sort_order=desc&pretty=true&offset=10",
"total" : 5169304
},
"result_content" : {
"data" : [ {
"data" : { ...
//truncated
"word_count" : 421,
"trending_score" : 6
} ...
// truncated
} ]
}
}Talkwalker Search Histogram API
https://api.talkwalker.com/api/v1/search/histogram/<type>How it works
With the Talkwalker Search Histogram API, you can retrieve the distribution of the number of search results for a given search query.
Histograms can be made for distribution over time or over specific metrics (number of comments, number of shares, reach, retweets etc.).
By setting min and max a histogram can be limited to a specific range (min_include and max_include control if those bounds are included).
interval defines the width of the bins, the accepted values are long integers for metrics or duration values (like 7d for 7 days) for published and search_indexed dates.
When using a bin size of entire days, timezone allows to set a timezone to specify the begin and end of the days.
Types
| type | Description |
|---|---|
|
Timestamp of publication (epoch time in milliseconds) |
|
Timestamp of indexation in Talkwalker (epoch time in milliseconds) |
|
The reach of an article/post represents the number of people who were reached by this article/post. |
|
The engagement of an article/post is the sum of actions made by others on that article/post. |
|
Number of Facebook shares an article has |
|
Number of Facebook likes an article has |
|
Number of Twitter retweets an article has |
|
Number of Twitter share an article has |
|
Number of Twitter followers a source has |
|
Number of YouTube views a video has |
|
Number of YouTube likes a video has |
|
Number of YouTube dislikes a video has |
|
Number of Comments an article has |
Parameters
| parameter | description | required? | allowed values | default value |
|---|---|---|---|---|
|
a read/write token specified in the API application |
required |
||
|
The query to search for |
required |
Talkwalker query syntax |
|
|
Minimum value for bins |
optional |
Long Integer value |
|
|
Maximum value for bins |
optional |
Long Integer value |
|
|
Include min value |
optional |
|
|
|
Include max value |
optional |
|
|
|
Bin Interval |
optional |
Long Integer (duration for |
|
|
Time zone (for interval) |
optional |
tz database timezone name ( i.e. `Europe/Luxembourg) |
|
|
Nested histogram |
optional |
|
|
|
Nested metric for time based histograms |
optional |
metric historgram types |
|
Possible values for interval when creating a histogram over published or search_indexed: year, quarter, month, week, day, hour, minute, second as well as numeric values with the units w (week), d (day), h (hours), m (minutes), and s (seconds). (e.g. 5d for 5 days or 2w for 2 weeks).
The maximum number of histogram bins is 400, if the min, max and interval parameters result in a larger number of bins, an error message (HTTP 400) is returned. Try reducing the range or increasing the interval.
value_type allows specifying a type for nested statistics per bin in a histogram over published or search_indexed.
Examples
Get a histogram over the last 8 days of online news results containing the word "birds"
Set the query to birds%20sourcetype:"ONLINENEWS". By default the Talkwalker Search Histogram API return results over the last seven days.
curl 'https://api.talkwalker.com/api/v1/search/histogram/published?access_token=demo&q=birds%20sourcetype:"ONLINENEWS"&interval=day&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v1/search/histogram?access_token=demo&q=birds%20sourcetype:\"ONLINENEWS\"&interval=7d&pretty=true",
"result_histogram" : {
"header" : {
"v" : [ "Number Results" ]
},
"data" : [ {
"t" : 1417478400000,
"v" : [ 4366.0 ]
}, {
"t" : 1417564800000,
"v" : [ 3385.0 ]
}, {
"t" : 1417651200000,
"v" : [ 4233.0 ]
}, {
"t" : 1417737600000,
"v" : [ 4071.0 ]
}, {
"t" : 1417824000000,
"v" : [ 2571.0 ]
}, {
"t" : 1417910400000,
"v" : [ 2191.0 ]
}, {
"t" : 1417996800000,
"v" : [ 3275.0 ]
}, {
"t" : 1418083200000,
"v" : [ 1140.0 ]
} ]
}
}Get a histogram with a resolution of 6 hours over the last 7 days of results containing the word "birds"
Set interval to 6h for 4 values per day.
curl 'https://api.talkwalker.com/api/v1/search/histogram/published?access_token=demo&q=birds&interval=6h'The interval parameter accepts the values year, quarter, month, week, day, hour, minute, second as well as numeric values with the units w (week), d (day), h (hours), m (minutes), and s (seconds).
Get a histogram over a specific range
Set min to 1390176000000 and max to 1390608000000 to get a histogram of results published between 20.01.2014 and 25.01.2014 with start timestamp included and end timestamp excluded (default values).
curl 'https://api.talkwalker.com/api/v1/search/histogram/published?access_token=demo&q=birds&min=1390176000000&max=1390608000000'The min and max parameters accept timestamps in epoch format (milliseconds after 1.1.1970 UTC).
Get a histogram and statistics over engagement
For types different from published and search_indexed, the histogram API also returns statistics (average, minimum, maximum and sum) over every bin.
(For published and search_indexed, you can specify an additional metric for statistics with the value_type parameter.)
curl 'https://api.talkwalker.com/api/v1/search/histogram/engagement?access_token=demo&q=birds&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v1/search/histogram/engagement?access_token=demo&q=birds&pretty=true",
"result_histogram" : {
"header" : {
"v" : [ "Number Results" ]
},
"data" : [ {
"v" : [ 333989.0 ],
"k" : 0.0,
"val" : [ {
"count" : 333989,
"min" : 0.0,
"max" : 80759.0,
"avg" : 22.01215608897299,
"sum" : 7351818.0
} ]
}, {
"v" : [ 5.0 ],
"k" : 82254.0,
"val" : [ {
"count" : 0,
"sum" : 0.0
} ]
}...
// truncated
{
"v" : [ 1.0 ],
"k" : 740286.0,
"val" : [ {
"count" : 1,
"min" : 822531.0,
"max" : 822531.0,
"avg" : 822531.0,
"sum" : 822531.0
} ]
} ]
}
}Talkwalker Search API and Talkwalker Projects
https://api.talkwalker.com/api/v1/search/p/<project_id>/resultsHow it works
Talkwalker users can use the topics defined in their project with the Talkwalker API. Topics can be used with the Search Results API and the Search Histogram API. This allows Talkwalker users to use the queries from their projects and to retrieve the documents they get in their Talkwalker project including changes and tags that were done in Talkwalker. In addition to the 30 days of search, the full history of Talkwalker projects is available in the search API, when used in combination with a Talkwalker project.
Parameters
| parameter | description | required? | default value |
|---|---|---|---|
|
API access token |
required |
|
|
The query to search for. |
required |
|
|
Number of results to skip (for paging) |
optional |
default: 0 |
|
Number of hits per page (for paging) |
optional |
default: 10 / maximum : 500 |
|
Criteria for sorting the results |
optional |
default: engagement |
|
Sorting order (ascending or descending) |
optional |
default: desc |
|
Turns highlighting on or off |
optional |
default: true |
|
One or more topics or panels that are defined in the Talkwalker project |
optional, multiple |
Get a list of all projects linked to an API application
Use the private access_token from your API application on the https://api.talkwalker.com/api/v1/search/info endpoint to get the list of all linked projects.
curl 'https://api.talkwalker.com/api/v1/search/info?access_token=<access_token>'Get a list of all resources
Resources are data retrieval settings from a Talkwalker project. This can be search-topics, filters, monitored-pages, source-panels, events, or saved-objects for for embedding in external tools.
To get a list of the resources defined in a Talkwalker project use the project_id and the access_token on the https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/resources endpoint.
curl 'https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/resources?access_token=<access_token>'Parameters
| parameter | description | required? | values |
|---|---|---|---|
|
a read/write token specified in the API application |
required |
|
|
filter on the type of resources |
optional |
search, filter, page, event, panel, savedobject |
|
filter on types of saved objects |
optional |
name of the saved-object type (name of the embedding destination) |
Example:
Get all saved objects from a project that were saved for embedding in an external tool called myapp.
curl 'https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/resources?access_token=<access_token>&type=sharedobject&object_type=myapp'Get a list of all tag IDs
To get a list of the tag IDs defined in a Talkwalker project use the project_id and the access_token on the https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/tags endpoint.
Tag IDs can be used in the query syntax.
curl 'https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/tags?access_token=<access_token>'Get search results and histograms for topics
The Project Search Result API https://api.talkwalker.com/api/v1/search/p/<project_id>/results and the the Project Search Histogram API https://api.talkwalker.com/api/v1/search/p/<project_id>/histogram can be used with the same parameters as the normal Search Result API and the Search Histogram API. Additionally to search a specific topic of a Talkwalker Project, set the parameter topic to one or more topic-IDs.
Talkwalker Streaming API V3
Overview
In general, the results obtained through the Talkwalker Streaming API depend on three factors: the dataset, the connection and the rules.
There are two datasets, which can be accessed:
-
Global data: Contains publicly available data
-
Project data: Consists of monitored and uploaded private data, which is unavailable outside of the project
The connection to the Streaming API can also be established in two ways:
-
Volatile: Data is only matched while the client is connected. Disconnection results in missing data.
→ Use case: Data is immediately consumed (e.g. a live dashboard) -
Persisted: By setting up a collector instead of a direct connection, the results are buffered for 7 days. Data is available in real-time, and contrary to volatile data, it is possible for a client to disconnect and reconnect without losing any data.
→ Use case: Handling a large amount of data and having a complete dataset. Generally recommended use case.
In the following sections, we will present 3 ways of accessing data:
-
Through stream: Volatile, global data
-
Through project: Volatile, project and global data
-
Through collector: Persisted, global data (and project data depending on setup)
A special use case of collectors is in accessing past data, which is presented separately in the final section.
Introduction
The following table shows parameters common to all endpoints that will be used in the Talkwalker Streaming API documentation.
| parameter | description | required? | default value |
|---|---|---|---|
|
a read/write token specified in the API application |
required |
|
|
receive a prettified JSON response if set to true |
optional |
false |
Stream IDs, rule IDs, collector IDs, etc used in these endpoints can only contain lowercase letters, numbers and the characters - and _.
They have to start with a lower case letter.
There are three types of chunks in the response of a request: CT_RESULT (containing the result), CT_CONTROL (containing information about the connection and possible information about the next result chunk) and CT_ERROR (for a possible error message).
Credits
Each result (independent on how many rules match) is counted as 1 credit.
If no credits are left, the data access is stopped and a control chunk containing the resume_offset of the end (needed for resuming) is sent.
API calls which don’t return any results are not counted.
The documents are billed after every completed chunk.
If a stream gets disconnected, a non-completed chunk will not be billed, since it needs to be restarted when resuming.
If an export task gets interrupted, all exported results are counted. Repeating the task means exporting duplicates, which count both times as 1 credit.
Project data access
When accessing data through a Talkwalker project, either a GET or POST HTTP request with several optional parameters can be used (see Table Optional parameters).
curl -XGET 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/results?access_token=<access_token>'curl -XPOST 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/results?access_token=<access_token>'
-H 'Content-Type: application/json; charset=UTF-8'
-d '{"q": "<query_string>"}'The POST request accepts a query in either its body or its URL.
The max_hits parameter is only accepted in the URL and stops the data access after a set amount of results.
| parameter | description | default value |
|---|---|---|
|
query to match (conjunctive) |
|
|
stops the data access after the given number of hits |
The result chunk for project data access has the form:
{
"chunk_type" : "CT_RESULT",
"chunk_result" : {
"data" : {
"data" : {
"url" : "<url>",
...
"matched" : {
"appearance" : "<appearance>"
},
...
}
}
}
}The url is not provided in every situation, e.g. Twitter results.
While new results count towards the max_hits parameter and require 1 credit each, it is also possible to obtain updated results.
The field appearance under matched provides this information. Updated results come at no cost.
Similar to when credits run out, only the specified maximum number of results is billed when the max_hits parameter is set and that amount of results is reached before the end of a chunk.
Still, the full chunk is delivered.
Streams
Receiving results from a stream works in two steps: First, the stream is created, then the data is accessed.
A stream is a collection of rules, where at least one must be provided for it to provide results. It is therefore recommended for the stream creation request to contain at least 1 rule in its body.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>?access_token=<access_token>'
-d '{"rules" : [{"rule_id" : "<rule_id>", "query":"<query>"}]}'
-H "Content-Type: application/json; charset=UTF-8"For the set of optional parameters for data access of a stream, consider Table Optional parameters.
curl -XGET 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/results?access_token=<access_token>&q=<query>'curl -XPOST 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/results?access_token=<access_token>'
-d '<query_data>'
-H 'Content-Type: application/json; charset=UTF-8'| parameter | description | default value |
|---|---|---|
|
the query to search for |
|
|
stops the stream after the given number of hits |
Each result consumes 1 credit, regardless of the number of matched rules.
When the parameter max_hits is set, only the specified maximum number of results will be billed, even if the entire timeframe gets streamed after reaching the limit.
Collectors
A collector allows users to define a hybrid collection of rules, based on: projects, streams and queries/filters. At least one of these needs to be defined.
All results matching the collector’s setup are buffered for 7 days on the server and consume 1 credit, each, no matter how many times they are read. In other words, data can be downloaded multiple times without additional cost.
A special use case of collectors lies in the access of past data, which is presented separately in the next section.
In this section, we present how to download the results of a collector and provide a list of operations (including the definition of a collector) along with examples.
Downloading the results of a collector
The search results of a collector can be accessed by a GET HTTP request, allowing for several optional parameters (see Table Optional parameters).
curl -XGET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?access_token=<access_token>'| parameter | description | possible values | default value |
|---|---|---|---|
|
position to resume the data access from. Can be retrieved from control chunks. |
"earliest" | "latest" | <resume_token> |
"earliest" |
|
what to do when we reached the most recent result |
"stop" | "wait" |
"wait" |
Operations on the definition of a collector
The Talkwalker Streaming API allows users to create/replace, retrieve and delete a collector, using the endpoint https://api.talkwalker.com/api/v3/stream/c/<collector_id>.
Definition of a collector
The definition of a collector consists of the following parameters.
| parameter | description | ||
|---|---|---|---|
|
|
||
|
|
|
|
|
|
||
|
|
|
|
|
|
||
|
|
|
|
|
|
||
|
|
||
The collector_query must never include both project and project_topics, only one of these two is allowed.
For all list parameters (streams, queries, topics in project_topics), only one element must be matched.
If multiple parameters are provided, they must all be matched.
{
"stream_id" : "stream-1",
"rules" : [{
"rule_id": "<rule_id>",
"query": "<query>"
}]
}{
"collector_id" : "collector-1",
"state" : "ACTIVE",
"collector_query" : {
"streams" : ["stream-1"],
"queries" : [{
"id" : "<query_id>",
"query" : "<query>"
}]
}
}In the response, the state of the collector is included, which can assume the following values: UNKNOWN, ACTIVE, ERROR, DELETED, PAUSED, NO_CREDITS.
Example
A new collector is created, whose collector_query includes a project and multiple queries.
curl -XPUT 'http://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {"project": "project-1", "queries":[{"id":"query-1", "query":"lang:en"}, {"id":"query-2", "query":"lang:fr"}]}}'
-H "Content-Type: application/json; charset=UTF-8"This collector collects all documents for project-1 that are either in English or in French.
Create/update a collector
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '<collector_definition>'
-H 'Content-Type: application/json; charset=UTF-8'For a <collector_definition>, the field state should not be set (it is set to ACTIVE automatically), and at least a project, a stream or a query must be set in the field collector_query.
A collector can include only one project but multiple queries and streams. The number of allowed queries and streams is not limited.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/collector-1?access_token=<access_token>&pretty=true'
-d '{"collector_query" : {"streams" : ["stream-1"], "queries" : [{"id" : "q-1", "query" : "lang:en"}]}}'
-H 'Content-Type: application/json; charset=UTF-8'{
"status_code" : "0",
"status_message" : "OK",
"request" : "PUT /api/v3/stream/c/collector-1?access_token=<access_token>&pretty=true",
"result_stream" : {
"collectors": [ {
"state": "ACTIVE",
"collector_id": "collector-1"
} ]
}
}Retrieve the definition of a collector
curl -XGET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v3/stream/c/collector-1?access_token=<access_token>&pretty=true",
"result_stream" : {
"collectors" : [{
"collector_id" : "collector-1",
"state" : "ACTIVE",
"query" : {
"streams" : ["stream-1"],
"queries" : [{
"id" : "q-1",
"query" : "lang:en"
}]
}
}]
}
}Delete a collector
Deleting a collector permanently removes it and its content. A new collector with the same name can be created, but it will not include the old collector’s results. Contrary, when updating a collector with a new query without deleting it, the old data is still included.
curl -XDELETE 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "DELETE /api/v3/stream/c/collector-1?access_token=<access_token>&pretty=true",
"result_stream" : {
"collectors" : [ {
"collector_id" : "collector-1",
"state" : "DELETED"
} ]
}
}Pause a collector
When calling this endpoint, a collector’s state changes to "PAUSED". A collector does not collect any real-time data while it is paused. When resuming a paused collector, all previously collected data is still included. A paused collector that is chosen as target for an export task still receives all exported data.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/pause?access_token=<access_token>&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "POST /api/v3/stream/c/collector-1/pause?access_token=<access_token>&pretty=true",
"result_stream" : {
"collectors" : [ {
"collector_id" : "collector-1",
"state" : "PAUSED"
} ]
}
}Resume a collector
Resuming a collector shifts its state from "PAUSED" to "ACTIVE". All incoming data from the point of resuming the collector onwards is stored again.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/resume?access_token=<access_token>&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "POST /api/v3/stream/c/collector-1/resume?access_token=<access_token>&pretty=true",
"result_stream" : {
"collectors" : [ {
"collector_id" : "collector-1",
"state" : "ACTIVE"
} ]
}
}Retrieve the information of all streams and collectors
curl -XGET 'https://api.talkwalker.com/api/v3/stream/info?access_token=<access_token>&pretty=true'{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v3/stream/info?access_token=<access_token>&pretty=true",
"result_stream" : {
"streams" : [ {
"stream_id" : "stream-1",
"enabled" : true
} ],
"collectors" : [ {
"collector_id" : "collector-1",
"state" : "ACTIVE"
} ]
}
}Past data export through collectors
Exports allow to create asynchronous tasks that copy a selection of past data into a collector, which can then be accessed through collectors.
There are three options to define the source of past data:
-
existing stream definitions (via a stream ID)
-
Talkwalker projects (via project ID)
-
specifying a query in the request body
An ongoing task can be checked or aborted by using its task ID, included in the response.
There are 3 POST endpoints, presented in the following subsections, which can execute an export task. These 3 endpoints share the following parameters, set inside the body:
| parameter | description | required? |
|---|---|---|
|
timestamp (milliseconds since 1.1.1970, e.g. 1539302400000) or date of the timeframe’s start (2018-10-12) |
required |
|
timestamp or date of the timeframe’s end |
optional |
|
ID of the collector |
required |
|
the query to search for (conjunctive to existing queries, i.e. matching all) |
optional |
Each exported result consumes 1 credit. Exporting the same result multiple times due to overlapping export tasks therefore requires multiple credits.
Creation of export tasks for Talkwalker projects
An export task for a Talkwalker project is started with a POST request to the /api/v3/stream/p/<project_id>/export endpoint.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>
-d '{"start": "<start>", "target":"<target>"}'Furthermore, for Talkwalker project export tasks, it is possible to further narrow down the result set.
If not the complete project but a selection of its topics must be matched, this can be specified by using the topics parameter.
| parameter | description | required? |
|---|---|---|
|
IDs of the topics taken into consideration |
optional |
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>
-d '{"start": "2018-11-15", "target":"testcollector", "topics":["topic_1","topic_2"]}'Creation of export tasks for existing streams
An export of data based on an existing stream definition is done, similar to projects, by sending a
POST request to the /api/v3/stream/s/<stream_id>/export endpoint.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/export?access_token=<access_token>
-d '{"start": <start>, "target":<collector_id>}'Creation of export tasks based on query parameter
With a third endpoint, it is possible to create an export task without providing a project or stream ID.
This endpoint depends on the query parameter, which consequently becomes required instead of optional.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/export?access_token=<access_token>
-d '{"start": <start>, "target":<collector_id>, "query":"<query>"}'Example
In this example, we wish to export all data from September 2018.
We start by creating an empty collector.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/collector-1?access_token=<access_token>'
-d '{"collector_query" : {}}'
-H "Content-Type: application/json; charset=UTF-8"The newly created collector is then used as target for the export task, where the time frame is limited to September 2018 using the start (as date) and stop (as timestamp) parameters in the request body.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/s/stream-1/export?access_token=<access_token>
-d '{"start": "2018-01-09", "stop":"1515366000", "target":"collector-1"}'{
"status_code" : "0",
"status_message" : "OK",
"request" : "POST /api/v3/stream/export?access_token=<access_token>",
"result_tasks" : {
"tasks" : [{
"creation_date" : "2018-14-31T15:24:34.069Z",
"type" : "export",
"id" : "task-1",
"status" : "queued",
"processed" : 0,
"progress" : 0.0,
"target" : "collector-1"
}]
}
}Best practice: If results for longer time periods shall be exported, it makes sense to split the export task into multiple smaller export tasks (e.g. one month when exporting results for half a year). This allows for a better estimation of the credit cost and the amount of results for the remaining time frame.
Status of an export
Using the task ID, which can be obtained from the response when creating a new task, the status of an export can be accessed.
curl -XGET 'https://api.talkwalker.com/api/v3/tasks/export/<task_id>?access_token=<access_token>{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v3/tasks/export/task-1?access_token=<access_token>",
"result_tasks" : {
"tasks" : [{
"creation_date" : "2018-03-21T08:23:00.335Z",
"type" : "export",
"id" : task-1,
"status" : "finished",
"processed" : 3,
"progress" : 1.0,
"target" : "coll-01"
}]
}
}The same request will give the list of all recent tasks if the task ID parameter is left aside.
curl -XGET 'https://api.talkwalker.com/api/v3/tasks/export?access_token=<access_token>{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v3/tasks/export?access_token=<access_token>",
"result_tasks" : {
"tasks" : [{
"creation_date" : "2018-03-21T08:28:35.469Z",
"type" : "export",
"id" : task-1,
"status" : "queued",
"processed" : 0,
"progress" : 0.0,
"target" : "collector-1"
},
{...},
{...}]
}
}Abort a task
Using the task ID a currently running export task can be aborted.
curl -XDELETE 'https://api.talkwalker.com/api/v3/tasks/export/<task_id>?access_token=<access_token>{
"status_code" : "0",
"status_message" : "OK",
"request" : "DELETE /api/v3/tasks/export/task-1?access_token=<access_token>",
"result_tasks" : {
"tasks" : [{
"id" : task-1,
"status" : "deleted"
}]
}
}Modifying documents with the Talkwalker API
Note: Any modifications of documents done via the Talkwalker API will overwrite changes done in Talkwalker. All earlier changes (manual or via export/import) in the same project are lost.
Single Documents
To change result documents, use the https://api.talkwalker.com/api/v2/docs/p/<project_id>/<operation> endpoint.
Creating new documents can be done on the create operation, updating documents is done with the update operation.
Deletion and un-deletion of documents can be done on the delete and undelete operations respectively.
The fields url, published, and content are required.
When left empty, some fields (for example source_type, post_type and lang) will be filled automatically with default values or automatically extracted values if possible.
Examples:
Create
When executing create for a document that already exists, the request fails and the existing document remains unchanged.
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/create?access_token=<access_token>' -d '
{
"url" : "http://www.example.com/docs/doc1.html",
"title" : "This is a title",
"content" : "Example content. Really not that much.",
"tags_marking" : ["read"],
"published" : "1430136532000"
}' -H 'Content-Type: application/json; charset=UTF-8'Update
When executing update and the document does not yet exist, the request fails and no new document is created.
important tag, and removing the read tag:curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/update?access_token=<access_token>' -d '
{
"url" : "http://www.example.com/docs/doc1.html",
"title" : "This is a new title",
"content" : "Example content. Really not that much.",
"+tags_marking" : ["important"],
"-tags_marking" : ["read"],
"extra_author_attributes" : {
"name" : null
},
"published" : "1430136532000"
}' -H 'Content-Type: application/json; charset=UTF-8'Fields that are of type array, can be updated in three ways: using "<fieldname>" to replace the whole array, "+<fieldname>" to add an item to the array, and "-<fieldname>" to remove an item.
Fields can be cleared by explicitly setting them null.
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/update?access_token=<application_access_token>' -d '
{
"url" : "http://www.news_site.com/news/news1.html",
"+customer_entities" : [ {
"type": "Category",
"id": ["Sports","Football"]
}, {
"type": "Place",
"id": ["USA", "Austin TX"]
} ]
}' -H 'Content-Type: application/json; charset=UTF-8'type is the entity-type (e.g. Person, Brand, Category etc), id is the actual entity name or hierarchy (e.g. Barack Obama, BMW, News etc).
Types are used for grouping entities in theme clouds, IDs are the displayed themes in the theme clouds.
Hierarchical IDs are defined as an array (the order is important!).
When multiple different entities have the same name (e.g. two persons with the same name), a unique identifier can be added after two underscores.
Max Mustermann__1, Max Mustermann__2 or Max Mustermann__politican. Only the part of the ID before the underscores will be displayed in the Talkwalker user interface.
The Talkwalker user interface will only show the first 2 levels of IDs.
Delete
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/delete?access_token=<access_token>' -d '
{
"url" : "http://www.example.com/docs/doc1.html"
}' -H 'Content-Type: application/json; charset=UTF-8'Undelete
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/undelete?access_token=<access_token>' -d '
{
"url" : "http://www.example.com/docs/doc1.html"
}' -H 'Content-Type: application/json; charset=UTF-8'Multiple Documents
Multiple documents can be manipulated using the https://api.talkwalker.com/api/v2/search/p/<project_id> endpoint.
The execution order of the given document operations is not guaranteed (multiple operations on a single document in a single request should be avoided).
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>?access_token=<access_token>' -d '
[{
"create": {
"url": "http://www.example.com/docs/doc1.html",
"title" : "This is the title of doc 1",
"content" : "and this is the content of doc 1",
}
}, {
"update": {
"url": "http://www.example.com/docs/doc2.html",
"title" : "This is the title of doc 2",
"content" : "and this is the content of doc 2",
}
}, {
"delete": {
"url": "http://www.example.com/docs/doc3.html"
}
}]' -H 'Content-Type: application/json; charset=UTF-8'If one or more operations fail the response will have the status code 49 and the response will include details of the failure.
The HTTP code of the response is still 200, even if some operations failed.This means that every document modification in the response needs to be checked separately for a (partial) failure. |
Parameters
| parameter | description | required? | values |
|---|---|---|---|
|
a read/write token specified in the API application |
required |
|
|
Specifies if the modified document should be returned |
optional |
hide (default), show |
When annotate is set to sentiment, entities or _all, the Talkwalker API recalculates sentiment or Talkwalker entities based on the updated content.
Please see the section Talkwalker Documents for a list of all writable fields that can be imported/modified with the Talkwalker API.
Custom Fields
In order to create a new document that includes custom fields, these custom fields have to be added to the project first (see Project Settings/Import Documents/Custom Metrics in the Talkwalker application).
We assume a project that has two custom fields: One decimal number double_1 and one String text_1.
A document like the following, including these two custom fields, can be created for this project:
curl -XPOST 'https://api.talkwalker.com/api/v2/docs/p/<project_id>/create?access_token=<access_token>' -d '
{
"url" : "http://www.example.com/docs/doc1.html",
"title" : "This is a match",
"content" : "Example content. Really not that much.",
"tags_marking" : ["read"],
"published" : "1430136532000",
"custom": {
"text_1" : "Example text. Also not that much.",
"double_1" : 0.75
}
}' -H 'Content-Type: application/json; charset=UTF-8'When matching a custom field, the result has the following form.
{
"chunk_type" : "CT_RESULT",
"chunk_result" : {
"data" : {
"data" : {
...
"custom" : {
"test_1" : "Example text. Also not that much."
}
},
"highlighted_data" : [ {
"title_snippet" : "This is a <b>match</b>",
"content_snippet" : "Example content. Really not that much.",
"matched" : {
"project_id" : "<project_id>",
"project_profiles" : [ {
"id" : "jl124veg_32fssfsd9238",
"type" : "topic",
"title" : "Match"
} ]
}
} ]
}
}
}Talkwalker Single Sign-on API
Overview
https://api.talkwalker.com/api/v2/auth/Note: The Single Sign-on API needs a special access token (of type authentication) and the endpoints must be called via a secure connection (HTTPS).
Talkwalker Login Url
curl 'https://api.talkwalker.com/api/v2/auth/u/<user_id>/loginurl?access_token=<access_token>&project_id=<project_id>'The Talkwalker Single Sign-on API is used to retrieve a single sign on URL for a Talkwalker account or application.
To get such an URL, the /loginurl endpoint is used,the returned login URL is only valid for 10 seconds.
The alternative endpoint api.talkwalker.com/api/v2/auth/loginurl?access_token=<access_token>&project_id=<project_id> can be used to login without specifying a user, the returned login url will authenticate as the account administrator.
Parameters
| parameter | description | required? | default value |
|---|---|---|---|
|
Authentication access token |
required |
|
|
ID of a Talkwalker project |
optional |
|
|
Overview page that will be opened on login |
optional |
¹ |
|
View that will be shown on login |
optional |
¹ |
|
Url the user will be redirected to on logout |
optional |
default login page |
|
Timeout for the generated login token |
optional |
10s |
|
Formatted json for testing |
optional |
false |
token_timeout accepts values in minutes or seconds (for example 5s or 1m) with a maximum time of 30m.
¹ Either page can be set (monitor, dashboard or home_screen) to lead the user to a overview page or view can be set to lead to a specific stored view (e.g. a specified dashboard).
If neither is provided the user will be lead to the home screen.
To get a list of all views see below.
https://api.talkwalker.com/api/v2/auth/u/<user_id>/loginurl?access_token=<access_token>&pretty=true&project_id=<project_id>{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v2/auth/u/<user_id>/loginurl?access_token=<access_token>&pretty=true&project_id=<project_id>",
"result_loginurl" : {
"single_sign_on_url" : "/app/login?login_token=<token>&user_id=<user_id>",
"user_id" : "<user_id>",
"expiration_date" : 1423064059056
}
}Logout
curl 'https://api.talkwalker.com/api/v2/auth/u/<user_id>/logout?access_token=<access_token>'The /logout-endpoint is used to log a user out from talkwalker and to invalidate all tokens that were created for this user.
All sessions for this user (either authenticated with a single sign on URL, or with a password) will be closed.
User List
curl 'https://api.talkwalker.com/api/v2/auth/users?access_token=<access_token>'This endpoint returns a list of all the users in an account and the projects they have access to.
https://api.talkwalker.com/api/v2/auth/users?access_token=<access_token>&pretty=true{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v2/auth/users?access_token=<access_token>&pretty=true",
"result_users" : {
"user" : [ {
"user_name" : "Admin 1",
"user_email" : user_1@site.com",
"user_id" : "user_id_1",
"project" : [ {
"project_id" : "project_id_1",
"project_name" : "Project 1",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"access_level" : "ACCOUNT_ADMIN"
}, {
"project_id" : "project_id_2",
"project_name" : "Project 2",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"access_level" : "ACCOUNT_ADMIN"
}, {
"project_id" : "project_id_3",
"project_name" : "Project 3",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"access_level" : "ACCOUNT_ADMIN"
} ]
}, {
"user_name" : "User 2",
"user_email" : user_2@site.com",
"user_id" : "user_id_2",
"project" : [ {
"project_id" : "project_id_2",
"project_name" : "Project 2",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"access_level" : "FULL_TOOL"
} ]
} ]
}
}Project List
curl 'https://api.talkwalker.com/api/v2/auth/projects?access_token=<access_token>'This endpoint returns a list of all the projects in an account and the users that have access.
https://api.talkwalker.com/api/v2/auth/projects?access_token=<access_token>&pretty=true{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v2/auth/projects?access_token=<access_token>&pretty=true",
"result_projects" : {
"project" : [ {
"project_id" : "project_id_1",
"project_name" : "Project 1",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"user" : [ {
"user_id" : "user_id_1",
"user_name" : "Admin 1",
"user_email" : user_1@site.com",
"access_level" : "ACCOUNT_ADMIN"
} ]
}, {
"project_id" : "project_id_2",
"project_name" : "Project 2",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"user" : [ {
"user_id" : "user_id_1",
"user_name" : "Admin 1",
"user_email" : user_1@site.com",
"access_level" : "ACCOUNT_ADMIN"
}, {
"user_id" : "user_id_2",
"user_name" : "User 2",
"user_email" : user_2@site.com",
"access_level" : "FULL_TOOL"
} ]
}, {
"project_id" : "project_id_3",
"project_name" : "Project 3",
"account_id" : "account_id_1",
"account_name : "account_name_1",
"user" : [ {
"user_id" : "user_id_1",
"user_name" : "Admin 1",
"user_email" : user_1@site.com",
"access_level" : "ACCOUNT_ADMIN"
} ]
} ]
}
}View List
curl 'https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/views?access_token=<access_token>'This endpoint returns a list of all the views in a project.
Note: This endpoint is part of the Talkwalker Project API and needs a read_write access token.
{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v2/talkwalker/p/<project_id>/views?access_token=<access_token>&pretty=true",
"result_views" : {
"projects" : [ {
"id" : "<project_id>",
"title" : "Project 1",
"dashboards" : [ {
"id" : "id1",
"title" : "Dashboard 1"
}, {
"id" : "id2",
"title" : "Dashboard 2"
}, {
"id" : "id3",
"title" : "Dashboard 3"
}, {
"id" : "id4",
"title" : "Dashboard 4"
}, {
"id" : "id5",
"title" : "Dashboard 5"
}, {
"id" : "id6",
"title" : "Dashboard 6"
} ]
} ]
}
}Talkwalker Image API
The Talkwalker Image API can be used to detect features or entities in images.
Images can be provided either as a URL or as a file-upload (a POST request containing multipart/form-data).
curl -XGET 'https://api.talkwalker.com/api/v2/detect/images/<type>?access_token=<access_token>&image_url=<image_url>'curl -XPOST 'https://api.talkwalker.com/api/v2/detect/images/<type>?access_token=<access_token>' -F 'image_file=@image.jpg'image_url is a URL-encoded URL (e.g. http%3A%2F%2Fwww.my-url.com%2Fimage.jpg`).
type defines the mode of the detection, the options are logo, object, scene.
The image_url parameter can only be used with specific URLs.
For example URLs that start with a specific prefix (http://www.my-url.com in the example).
Please contact Talkwalker to have your account configured.
|
{
"status_code": "0",
"status_message": "OK",
"request": "GET /api/v2/detect/images/logo?access_token=<access-token>&image_url=http%3A%2F%2Fwww.my-url.com%2Fimage.jpg",
"result_processing": {
"match": [
{
"top": 848,
"left": 678,
"right": 983,
"bottom": 1155,
"confidence": 0.9657765030860901,
"id": "starbucks"
},
{
"top": 880,
"left": 1520,
"right": 1826,
"bottom": 1209,
"confidence": 0.9366655349731445,
"id": "starbucks"
},
{
"top": 672,
"left": 239,
"right": 521,
"bottom": 916,
"confidence": 0.9254658222198486,
"id": "starbucks"
}
],
"height": 1365,
"width": 2048
}
}By adding the parameter detect, one or multiple Talkwalker image IDs can be
specified to restrict the detection to only these.
match contains the list of matched logos, each with their Talkwalker image ID and a confidence value from 0 to 1.
If the selected type is logo, the position of the match is given by the top, left, right and bottom corners of the bounding box.
Additional logos, scenes and objects can be added on request.
Talkwalker Channelmonitoring API
Channelmonitoring suggest
This provides the same functionality as the pagemonitoring suggest in the talkwalker. Given a string (url, name, …) and a type (default = auto), it will provide several candidates.
curl -XGET https://api.talkwalker.com/api/v2/talkwalker/p/<projectid>/monitoring/suggest?input=<url/string>&type=auto&access_token=<access_token>{
"status_code" : "0",
"status_message" : "OK",
"request" : "...",
"result_monitoring_pages" : {
"data" : [ {
"title" : "ABC",
"type" : "page",
"access_url" : "http://talkwalker.com/296043200790",
"query" : "channel:\"vtwqablxreaaaacgbieemqkdivbe6t2lcicgmzlfmqnci2duorydulzpo53xonf4zdsnrqgqztembqg44ta\""
}, ... ]
}
}Fetch query
Input: the access_url and the site monitoring type
Output: query to be used in stream
curl -XGET https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/monitoring/fetch?type=type_id&access_url=http%%3A%%2F%%2Ftalkwalker.com%%2FABC&access_token=<access_token>{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v2/talkwalker/p/<project_id>/monitoring/fetch?type=type_id&access_url=http%%3A%%2F%%2Ftalkwalker.com%%2FABC&access_token=<access_token>",
"result_monitoring_pages" : {
"data" : [{
"title" : "ABC",
"type" : "type_id",
"access_url" : "http://talkwalker.com/ABC",
"query" : "channel:\"vtwqablxreaaaacgbieemqkdivbe6t2lcicgmzlfmqnci2duorydulzpo53xonf4zdsnrqgqztembqg44ta\""
}]
}
}Talkwalker Query Syntax
A single search query can support up to 50 operands and be up to 4096 characters long in length. To create complex queries, operands may be combined using Boolean Operators.
All queries are executed in their unaccented and case insensitive form, thus a search for "éléVE" will also match all documents with the word "eleve". No language stemming is being done, thus a search for the "children" won’t return results with the word "child". Non Latin characters and Emojis can also be used for creating queries.
Special Transformations
These transformations apply when a query contains no operators from the query syntax (quotes, AND, OR, wildcards etc, see below).
Words with only capital letters (and special chars +-&) are executed as exact (case sensitive) raw data search (ABC = ++"ABC", A&B = ++"A&B").
Screen names (@name), hashtags (#hashtag), cashtags ($cashtag) as well as words containing a dash (-), a plus (+) or an ampersand (&) are executed as (case insensitive) raw data search (@username = +"@username" , p&t = +"p&t").
If a query contains multiple simple words (no special characters like (#@+-&), no operators and is not only capital letters, it is executed as a proximity search.
The maximum number of jumps is set to (#words - 1) * 10 (cat dog mouse bird = "cat dog mouse bird"~30).
To prevent this behaviour use the explicit query syntax below. (instead of cat dog mouse use cat OR dog OR mouse, cat AND dog AND mouse or "cat dog mouse" to search for one of the words, all the words or the exact phrase.
Boolean Operators
AND |
|
|
AND NOT |
|
|
OR |
|
|
Exclusion of Keywords |
Negative filters can be created by using the operator |
|
Phrase Search |
Quotes |
|
Combinations |
Brackets |
|
Wildcard Search |
The Wildcard operator |
|
Wildcard Search – one character |
The question mark |
|
Proximity Search |
The tilde symbol |
|
Fuzzy X Search |
The tilde symbol |
|
Fuzzy Search |
The tilde symbol |
|
Raw Data Search |
A simple |
|
Exact Raw Data Search |
Two |
|
NEAR/x |
The |
|
ONEAR/x |
Same as |
|
Sentence Search |
The SENTENCE operator works similar to the NEAR/x operator. It searches for keywords that appear in the same sentence. SENTENCE can also be used with multiple terms. |
|
Ordered Sentence Search |
Same as |
|
Note:
In phrase search and raw data phrase search ("" or +"") the number and type white space characters are ignored.
For example "BMW series" (one space) will also match documents which contain "BMW series" (two spaces) and vice versa.
White space characters include spaces, tabs and new line characters, also transitions between letters and special characters are considered as whitespace.
For example +"P&T" will match P&T but also P& T and P & T.
Advanced Search Options:
Single Keyword Search |
Search for simple brands, products, keywords, etc. |
|
Title Search |
It searches within the title of an article. |
|
Content Search |
It searches within the article |
|
Author Search |
It searches for authors of articles. |
|
Author Short Name Search |
Search for a specific author short name (case sensitive). |
|
Mention Search |
Search for a mention of a user (case sensitive). |
|
Hashtag Search |
Search for a hashtag (case sensitive). |
|
Language Search |
It searches for languages of articles. |
|
Author Gender Search |
Search only for |
|
Author ID Search |
Search for posts by the author with a certain ID. |
|
Source ID Search |
Search for posts, that were released at the source with a certain ID. |
|
Author Description Search |
Search in the description (biography) of the author with |
|
Source Country Restriction |
It searches for the country of origin of sources. |
|
Author Country Restriction |
Search for articles by authors from a specific country, |
|
Source Type Restriction |
|
|
Comments Search |
Find only comments by setting |
|
Retweets Search |
Find only retweets with |
|
Twitter Reply Search |
Find only tweets that are replies to other tweets |
|
Verified Author Search |
Find only documents from verified authors |
|
Verified Source Search |
Find only documents from verified sources |
|
Questions Search |
Search for questions. |
|
Image Search |
|
|
Audio Search |
|
|
Video Search |
|
|
Talkwalker Tags Search |
|
|
Score Search |
|
|
Post Type Search |
|
|
Device Search |
Find only documents from certain devices. Possible values are BOT MOBILE_IOS MOBILE_ANDROID MOBILE_WINDOWS MOBILE_BLACKBERRY MOBILE_OTHER TABLET_IOS TABLET_ANDROID TABLET_WINDOWS TABLET_OTHER PC OFFICIAL_WEBSITE EXTERNAL_WEBSITE. |
|
Image Search |
|
|
Demographics Search |
|
|
Customer Entity Search |
|
|
Custom Field Search |
|
|
For sourcetype, posttype, lang, sourcegeo, image (objects and scenes) and demographics (familiy-status, occupation and interest) see Value options or use the suggestions in the query editor.
Brand related images (logos) and cities are only suggested by the query editor.
Url based Search
Url Search |
|
|
Parent Url Search |
|
|
Host Url Restriction |
|
|
Domain Url Restriction |
|
|
Site Search |
|
|
In Urls Search |
|
|
All URL based search features are case sensitive.
Metric (Minimum / Maximum) Restrictions
metric_name:>n , metric_name:<n and metric_name:n return only documents which match a specific value or range of a metric.
Following tables explains the possible metrics
| metric_name | Description | Example |
|---|---|---|
|
The reach of an article/post represents the number of people who were reached by this article/post. |
|
|
The engagement of an article/post is the sum of actions made by others on that article/post. |
|
|
Number of views a Dailymotion video has |
|
|
Number of Facebook share an article has |
|
|
Number of Facebook likes an article has |
|
|
Number of Twitter retweets an article has |
|
|
Number of Twitter share an article has |
|
|
Number of Twitter followers a source has |
|
|
Number of Twitter likes an article has |
|
|
Number of YouTube views a video has |
|
|
Number of YouTube likes a video has |
|
|
Number of YouTube dislikes a video has |
|
|
Number of circulated copies of the source |
|
|
Number of printed copies of the source |
|
|
Number of sold copies of the source |
|
|
Number of Instagram likes a post has |
|
|
Number of Instagram followers a post has |
|
|
Number of Comments an article has |
|
|
Timestamp of publication (epoch time in milliseconds) |
|
|
Timestamp of indexation in Talkwalker (epoch time in milliseconds) Note: The metric |
|
|
Get a random sample of the results (percent of the total number of results i.e. setting 25 will return one of four the documents) values:1-100 |
|
|
Similar to sample, with higher precision (i.e. setting 2000 will return one of 500 documents) values:1-1000000 |
|
|
Porn level of an article (0-100) |
|
|
Spam level of an article (0-100) |
|
|
The detected sentiment of the article (values -5 (negative) to 5 (positive)). |
|
|
The number of words in the content |
|
Geographic Restrictions
Note: Some documents have precise geographic data in form of GPS measured coordinates provided by the source. For other documents this data is based on source metadata, with a certain precision level.
These levels (ordered from lowest precision to highest) are: country, region and city (extracted data) and coordinates (exact data).
The coordinates for lower precision geographic data are equal to their capital.
Geographic restrictions exist for
sourcegeo : location of the source of the articles (e.g. a page, a site, a publication).
authorgeo : general location of the author of the article.
articlegeo : location where the article was written.
| Restriction | Description | Example |
|---|---|---|
|
Restricts the results to a geographic area defined either by coordinates ( |
|
|
Restricts to documents that have a minimum precision level of location data. Possible levels are |
|
|
Restricts to documents that have a specific geo detection accuracy. Options are |
|
Example: Search for documents that are in a box that roughly corresponds to Luxembourg and have exact coordinates.
Luxembourg’s north end is at around 50.3°, south is at 49.4°, west at 5.7° and east at 6.5°, the upper left corner is 50.3,5.7 the lower right corner is 49.4,6.5. The final query is :
sourcegeo:50.3,5.7;49.4,6.5 AND sourcegeo_resolution:coordinates.
Special Query Modifiers
All queries are executed in their unaccented and case insensitive form on the content and the title of documents. To change this behaviour, use flag:<modifier_name> to enable special query modes.
| Modifier Name | Description | Example |
|---|---|---|
|
Query will also match URLs and links. |
|
|
Query will also match author field |
|
|
Query will also match author description field |
|
|
Use Raw data search as default. All keywords are considered as case-insensitive exact character string including special characters and punctuation. |
|
|
Use Exact raw data search as default. All keywords are considered as case-insensitive exact character string including special characters and punctuation. |
|
|
Use Fuzzy Search as default. All keywords will also match combined words |
|
The special modifiers can be combined: carsharing flag:matchauthor flag:matchfuzzywords searches for words like carsharing, car sharing or car-sharing in the fields title, content and author_name.
Note: When matchinurls or matchauthor is set, API results will not have highlighting in snippets when one of these fields is matched.
Talkwalker Documents
Fields
| field_name | datatype | accepted dataformat | required | writable | default | description | Example |
|---|---|---|---|---|---|---|---|
url |
string |
url¹ |
yes |
yes |
- |
URL |
|
published |
long |
timestamp in ms |
yes |
yes |
- |
Published |
|
title |
string |
<500 chars |
yes |
yes |
- |
Title |
|
content |
string |
<50,000 chars |
yes |
yes |
- |
Content |
|
indexed |
long |
- |
no |
no |
- |
Indexed |
|
search_indexed |
long |
- |
no |
no |
- |
Search Indexed |
|
title_snippet |
string |
- |
no |
no |
- |
Title Snippet |
|
content_snippet |
string |
- |
no |
no |
- |
Content Snippet |
|
root_url |
string |
- |
no |
no |
extracted from url |
Root URL |
|
domain_url |
string |
- |
no |
no |
extracted from url |
Domain URL |
|
host_url |
string |
- |
no |
no |
extracted from url |
Host URL |
|
parent_url |
string |
- |
no |
no |
- |
Parent URL |
|
lang |
string |
2 char iso |
no |
yes |
detected from content |
Language of the Article |
|
porn_level |
integer |
0..100 |
no |
yes |
- |
Pornography Level |
|
fluency_level |
integer |
0..100 |
no |
yes |
- |
Fluency Level |
|
spam_level |
integer |
0..100 |
no |
yes |
- |
Spam Level |
|
sentiment |
integer |
-5..5 |
no |
yes |
0 |
Sentiment |
|
reach |
integer |
>0 |
no |
yes |
- |
Reach |
|
engagement |
integer |
>0 |
no |
yes |
- |
Engagement |
|
provider |
string |
a-z0-9_ <100 chars |
no |
yes |
- |
Document Provider Name |
|
source_type |
list of string |
see list² |
no |
yes |
"OTHER" |
Source Type |
|
post_type |
list of string |
see list² |
no |
yes |
"TEXT" |
Post Type |
|
cluster_id |
string |
- |
no |
no |
- |
Cluster Id |
|
meta_cluster_id |
string |
- |
no |
no |
- |
Meta Cluster Id |
|
tags_internal |
list of string |
- |
no |
no |
- |
Internal Tags 1 |
|
tags_marking |
list of string |
see list² |
no |
yes |
- |
Internal Tags 2 |
|
tags_customer |
list of string |
see ³ |
no |
yes |
- |
Customer Tags |
|
tags_plugin |
list of string |
see ³ |
no |
yes |
- |
Plugin tags |
|
matched_query |
string |
no |
no |
- |
Matched Query |
||
matched_profile |
string |
no |
no |
- |
Matched Profile |
||
images |
list of image |
see below |
no |
see below |
- |
Images |
|
videos |
list of video |
see below |
no |
see below |
- |
Videos |
|
article_extended_attributes |
article_extended_attributes |
see below |
no |
see below |
- |
Article Attributes |
|
source_extended_attributes |
source_extended_attributes |
see below |
no |
see below |
- |
Source Attributes |
|
extra_article_attributes |
extra_article_attributes |
see below |
no |
see below |
- |
Article Attributes |
|
extra_author_attributes |
extra_author_attributes |
see below |
no |
see below |
- |
Author Attributes |
|
extra_source_attributes |
extra_source_attributes |
see below |
no |
see below |
- |
Source Attributes |
|
customer_entities |
list of customer_entity |
see below |
no |
see below |
- |
Customer Entities |
See the chapter on Protocols, Encodings and Value Field Options for possible values for the fields sourcetype, lang, or geo.
¹ Can not be changed after creating a new document.
² See list of value options.
³ tags_customer: a-zA-Z0-9- or space, supports hierarchy using /, can only be set in project specific documents, not in general document import.
` tags_plugin` : have to be in the form <vendor_id><vender_field>:<value>
Content
Talkwalker provides result snippets for all content. In all cases, the content
field only contains the first words of the document, in addition, we provide
the part of the document which matches the query in the content_snippet field.
In the Streaming API a snippet is provided for every matching rule.
URLs
To filter on specific websites in a query, the fields domain_url and host_url can be used.
host_url is used for specific hosts like www.talkwalker.com or blog.talkwalker.com, while domain_url would filter on all host in a specific domain
(i.e. domain_url:blog.talkwalker.com would return all results of the domain talkwalker.com also those from www.talkwalker.com while host_url:blog.talkwalker.com would return only results
from blog.talkwalker.com not from www.talkwalker.com).
Sentiment
Talkwalker uses natural language processing (NLP) to compute a general sentiment for the documents in our index. The accuracy of automatic detection is limited by irony, sarcasm and misspellings in the documents. Sentiment analysis is available for:
| Language | Language Code | Language | Language Code |
|---|---|---|---|
|
sq |
|
hu |
|
ar |
|
it |
|
zh_cn, zh_tw |
|
ko |
|
hr |
|
ms |
|
cs |
|
no |
|
da |
|
pl |
|
nl |
|
pt |
|
en |
|
ru |
|
fi |
|
sk |
|
nl |
|
es |
|
fr |
|
sv |
|
de |
|
tr |
Reach
The reach of an article/post represents the number of people who were reached by this article/post.
Note that the views only get set to a proper value if the host of the URL is either a domain (like theguardian.com) or if it is a domain with a well-known 3rd-level-subdomain in front (mainly applies to www, e.g. www.theguardian.com).
Reach is set to 0 for other hosts, i.e. hosts with other 3rd-level-subdomains, like on foobar.blogspot.com, as using the Alexa views of the domain would assign much too high reach to mere sub-hosts otherwise.
For imported documents reach can be set via the Talkwalker API.
Reach is calculated in the following ways:
Blogs; News Sites; Forums: Number of Page Views
Facebook: The Number of Fans of the Page (Note: Only available for public pages, which are monitored by Talkwalker, we don’t collect any fan counts for user profiles)
Twitter: The number of Followers of the author
image object
| field_name | datatype | accepted dataformat | required | writable | default | description | Example |
|---|---|---|---|---|---|---|---|
url |
string |
normalized url |
yes |
yes |
- |
Url of Image |
|
legend |
string |
<1000 chars |
no |
yes |
- |
Legend Text |
|
width |
integer |
no |
yes |
- |
Width of image |
||
height |
integer |
no |
yes |
- |
Height of image |
"images" : [{
"url" : "http://www.example.com/image1.jpg",
"legend" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit"
}]video object
| field_name | datatype | accepted dataformat | required | writable | default | description | Example |
|---|---|---|---|---|---|---|---|
url |
string |
normalized url |
yes |
yes |
- |
Url of Video |
|
legend |
string |
<1000 chars |
no |
yes |
- |
Legend Text |
|
width |
integer |
no |
yes |
- |
Width of video |
||
height |
integer |
no |
yes |
- |
Height of video |
"videos" : [{
"url" : "http://www.example.com/video1.mpg",
"legend" : "Lorem ipsum dolor sit amet, consectetur adipiscing elit"
}]customer_entity object
| field_name | datatype | accepted dataformat | required | writable | default | description | Example |
|---|---|---|---|---|---|---|---|
type |
string |
person, place, organization, keyword, iptc, source_distribution_type |
yes |
yes |
- |
Entity Type |
|
id |
list of string (ordered) |
<100 chars |
yes |
yes |
- |
Entity Id |
|
The Talkwalker user interface will only show the first 2 levels of Ids.
The list of Ids defines a hierarchy where each Id defines a superset of the following Ids.
See list of id options for source_distribution_type in chapter "formats".
Examples:
"customer_entities" : [{
"type": "Place",
"id": ["USA", "Austin TX"]
}]"customer_entities" : [{
"type": "organization",
"id": ["UNO"]
}, {
"type": "organization",
"id": ["WHO"]
}]Attributes
These fields are only set for certain post types.
Article extended attributes fields will be updated for up to 1 month.
The source extended attributes represent the exact value at publication.
Not all urls will have all meta data, e.g.:
-
Blog, news and messageboard posts (not their comments), will only have facebook_shares, twitter_shares set.
-
All the other types will only be set if the sourcetype is of the same type and if the data is available.
article_extended_attributes object
| field_name | datatype | accepted dataformat | required | writable | default | description |
|---|---|---|---|---|---|---|
facebook_shares |
long |
>0 |
no |
yes |
- |
Article Facebook Shares |
facebook_likes |
long |
>0 |
no |
yes |
- |
Article Facebook Likes |
twitter_retweets |
long |
>0 |
no |
yes |
- |
Article Twitter Retweets |
twitter_likes |
long |
>0 |
no |
yes |
- |
Article Twitter Likes |
url_views |
long |
>0 |
no |
yes |
- |
Article URL Views |
pinterest_likes |
long |
>0 |
no |
yes |
- |
Article Pinterest Likes |
pinterest_pins |
long |
>0 |
no |
yes |
- |
Article Pinterest Pins |
pinterest_repins |
long |
>0 |
no |
yes |
- |
Article Pinterest Re-Pins |
youtube_views |
long |
>0 |
no |
yes |
- |
YouTube Video Views |
youtube_likes |
long |
>0 |
no |
yes |
- |
YouTube Video Likes |
youtube_dislikes |
long |
>0 |
no |
yes |
- |
YouTube Video Dislikes |
linkedin_shares |
long |
>0 |
no |
yes |
- |
Article Linkedin Shares |
linkedin_likes |
long |
>0 |
no |
yes |
- |
Article Linkedin Likes |
dailymotion_views |
long |
>0 |
no |
yes |
- |
Dailymotion Video Views |
vkontakte_likes |
long |
>0 |
no |
yes |
- |
Vkontakte Article Likes |
vkontakte_shares |
long |
>0 |
no |
yes |
- |
Vkontakte Article Shares |
instagram_likes |
long |
>0 |
no |
yes |
- |
Instagram Image Likes |
twitter_shares |
long |
>0 |
no |
yes |
- |
Article Twitter Shares |
source_extended_attributes object
| field_name | datatype | accepted dataformat | required | writable | default | description |
|---|---|---|---|---|---|---|
alexa_pageviews |
long |
>0 |
no |
yes |
- |
Alexa Page Views |
facebook_followers |
long |
>0 |
no |
yes |
- |
Facebook Followers |
twitter_followers |
long |
>0 |
no |
yes |
- |
Twitter Followers |
instagram_followers |
long |
>0 |
no |
yes |
- |
Instagram Followers |
pinterest_followers |
long |
>0 |
no |
yes |
- |
Pinterest Followers |
linkedin_followers |
long |
>0 |
no |
yes |
- |
Linkedin Followers |
print_run_print |
long |
>0 |
no |
yes |
- |
Print Run Print |
print_run_sales |
long |
>0 |
no |
yes |
- |
Print Run Sales |
print_run_circulation |
long |
>0 |
no |
yes |
- |
Print Run Circulation |
extra_article_attributes object
| field_name | datatype | accepted dataformat | required | writable | default | description |
|---|---|---|---|---|---|---|
world_data/continent |
string |
- |
no |
no |
Article Continent |
|
world_data/country |
string |
- |
no |
no |
Article Country |
|
world_data/region |
string |
- |
no |
no |
Article Region |
|
world_data/city |
string |
- |
no |
no |
Article City |
|
world_data/longitude |
double |
- |
no |
no |
Article Longitude |
|
world_data/latitude |
double |
- |
no |
no |
Article Latitude |
|
world_data/country_code |
string |
- |
no |
no |
||
world_data/resolution |
string |
- |
no |
no |
||
geo |
string |
see list² |
no |
yes |
Article location |
|
id |
string |
- |
no |
no |
Article ID |
|
type |
string |
- |
no |
no |
Article Type |
|
name |
string |
- |
no |
no |
Article Name |
|
gender |
string |
- |
no |
no |
Article Gender |
|
image_url |
string |
- |
no |
no |
Article Image URL |
|
short_name |
string |
- |
no |
no |
Article Short Name |
|
url |
string |
- |
no |
no |
Article URL |
geo is a write only field that is used to set the world_data fields and can not be read afterwards.+
Please see the geo_id API document for a list of possible values.
extra_author_attributes object
| field_name | datatype | accepted dataformat | required | writable | default | description |
|---|---|---|---|---|---|---|
world_data/continent |
string |
- |
no |
no |
Author Continent |
|
world_data/country |
string |
- |
no |
no |
Author Country |
|
world_data/region |
string |
- |
no |
no |
Author Region |
|
world_data/city |
string |
- |
no |
no |
Author City |
|
world_data/longitude |
double |
- |
no |
no |
Author Longitude |
|
world_data/latitude |
double |
- |
no |
no |
Author Latitude |
|
world_data/country_code |
string |
- |
no |
no |
||
world_data/resolution |
string |
- |
no |
no |
||
geo |
string |
see list² |
no |
yes |
Author location |
|
id |
string |
- |
no |
no |
Author ID |
|
type |
string |
- |
no |
no |
Author Type |
|
name |
string |
- |
no |
yes |
Author Name |
|
gender |
string |
- |
no |
yes |
Author Gender |
|
image_url |
string |
url |
no |
no |
Author Image URL |
|
short_name |
string |
no space allowed |
no |
yes |
Author Short Name |
|
url |
string |
url |
no |
yes |
Author URL |
geo is a write only field that is used to set the world_data fields and can not be read afterwards.+
Please see the geo_id API document for a list of possible values.
extra_source_attributes
| field_name | datatype | accepted dataformat | required | writable | default | description |
|---|---|---|---|---|---|---|
world_data/continent |
string |
- |
no |
no |
Source Continent |
|
world_data/country |
string |
- |
no |
no |
Source Country |
|
world_data/region |
string |
- |
no |
no |
Source Region |
|
world_data/city |
string |
- |
no |
no |
Source City |
|
world_data/longitude |
double |
- |
no |
no |
Source Longitude |
|
world_data/latitude |
double |
- |
no |
no |
Source Latitude |
|
world_data/country_code |
string |
- |
no |
no |
||
world_data/resolution |
string |
- |
no |
no |
||
geo |
string |
see list² |
no |
yes |
Source location |
|
id |
string |
- |
no |
no |
Source ID |
|
type |
string |
- |
no |
no |
Source Type |
|
name |
string |
- |
no |
yes |
Source Name |
|
gender |
string |
- |
no |
no |
Source Gender |
|
image_url |
string |
url |
no |
yes |
Source Image URL |
|
short_name |
string |
- |
no |
no |
Source Short Name |
|
url |
string |
url |
no |
yes |
Source URL |
geo is a write only field that is used to set the world_data fields and can not be read afterwards.+
Please see the geo_id API document for a list of possible values.
Evolution and stability of document fields
The structure of the documents will not be changed. Existing fields will not be removed and their formatting will not be changed. Occasionally, new fields will be added to the documents and the order of fields can change, please take this into account when implementing a custom client.
Streaming
(repeated extra entries for each matching rule, available in streaming only)
highlighted_data object
On streaming, this information is present in an highlighted_data object
| field | Name | Write access through API | Comment |
|---|---|---|---|
|
matched rule |
- |
ID of matched rule |
|
matched rule |
- |
Query of matched rule (when id is not set) |
|
matched stream |
- |
ID of matched stream |
|
matched panel |
- |
ID of matched panel |
|
matched project |
- |
ID of matched Talkwalker project |
|
Matched Profiles |
- |
Profiles which matched (if Talkwalker project) |
|
Title Snippet |
- |
If a match occurred in the title, this field will contain the snippet related to the query set in the datafeed. |
|
Content Snippet |
- |
If a match occurred in the article, this field will contain the snippet related to the query set in the datafeed. |
Protocols, Encodings and Value Field Options
Protocols and Encodings
The Talkwalker API uses HTTP protocol 1.1. The Streaming API streams documents using the HTTP 1.1 Chunked transfer encoding mechanism.
The data is compressed using gzip: "Accept-Encoding:gzip" must be set in the header.
The Encoding used is UTF-8.
The maximum size for POST and PUT requests is 5120 kB.
Evolution of JSON fields
The structure of the json responses will not be changed. Existing fields will not be removed and their formatting will not be changed. However, new fields will be added to the responses and the order of fields can change, please take this into account when implementing a custom client.
Value options
The Following tables contain possible options and formats for certain fields.
Source Type Options
| Media Source Types | |
|---|---|
|
All news sites |
|
Printed magazines online sites |
|
Printed newspaper online sites |
|
Results from sites that publish press releases |
|
Results from TV or radio station sites |
|
News agency sites |
|
News results that do not fall under of the other news categories |
|
All print articles |
|
Articles from printed magazines |
|
Articles from printed newspapers |
|
Other printed articles |
|
Newsletters |
|
All blog sites |
|
All forums and message boards |
|
All social media sites |
|
Results from Twitter |
|
Results from Facebook |
|
Results from YouTube |
|
Results from LinkedIn |
|
Results from Google+ |
|
Results from Flickr |
|
Results from Foursquare |
|
Results from Instagram |
|
Results from Pinterest |
|
Results from Mixcloud |
|
Results from SoundCloud |
|
Results from Vimeo |
|
Results from Dailymotion |
|
Results from Weibo |
|
Results from vk.com |
|
Results from Vine |
|
All articles from broadcast |
|
Articles from radio |
|
Articles from TV |
|
Everything else which does not fit into the above listed categories |
Language Options
ABKHAZIAN |
|
HERERO |
|
PALI |
|
AFAR |
|
HINDI |
|
PANJABI |
|
AFRIKAANS |
|
HIRI MOTU |
|
PERSIAN |
|
AKAN |
|
HUNGARIAN |
|
POLISH |
|
ALBANIAN |
|
ICELANDIC |
|
PORTUGUESE |
|
AMHARIC |
|
IDO |
|
PUSHTO |
|
ARABIC |
|
IGBO |
|
QUECHUA |
|
ARAGONESE |
|
INDONESIAN |
|
RAETO ROMANCE |
|
ARMENIAN |
|
INTERLINGUA |
|
ROMANIAN |
|
ASSAMESE |
|
INTERLINGUE |
|
RUNDI |
|
AVARIC |
|
INUKTITUT |
|
RUSSIAN |
|
AVESTAN |
|
INUPIAQ |
|
SAMOAN |
|
AYMARA |
|
IRISH |
|
SANGO |
|
AZERBAIJANI |
|
ITALIAN |
|
SANSKRIT |
|
BAMBARA |
|
JAPANESE |
|
SARDINIAN |
|
BASHKIR |
|
JAVANESE |
|
SCOTTISH GAELIC |
|
BASQUE |
|
KANNADA |
|
SERBIAN |
|
BELARUSIAN |
|
KANURI |
|
SHONA |
|
BENGALI |
|
KASHMIRI |
|
SICHUAN YI |
|
BIHARI |
|
KAZAKH |
|
SINDHI |
|
BISLAMA |
|
KHMER |
|
SINHALESE |
|
BOSNIAN |
|
KIKUYU |
|
SLOVAK |
|
BRETON |
|
KINYARWANDA |
|
SLOVENIAN |
|
BULGARIAN |
|
KIRGHIZ |
|
SOMALI |
|
BURMESE |
|
KOMI |
|
SOUTHERN SOTHO |
|
CATALAN |
|
KONGO |
|
SOUTH NDEBELE |
|
CHAMORRO |
|
KOREAN |
|
SPANISH |
|
CHECHEN |
|
KURDISH |
|
SUNDANESE |
|
CHINESE |
|
KWANYAMA |
|
SWAHILI |
|
CHINESE SIMPLIFIED |
|
LAO |
|
SWATI |
|
CHINESE TRADITIONAL |
|
LATIN |
|
SWEDISH |
|
CHURCH SLAVIC |
|
LATVIAN |
|
TAGALOG |
|
CHUVASH |
|
LIMBURGISH |
|
TAHITIAN |
|
CORNISH |
|
LINGALA |
|
TAJIK |
|
CORSICAN |
|
LITHUANIAN |
|
TAMIL |
|
CREE |
|
LUBA KATANGA |
|
TATAR |
|
CROATIAN |
|
LUXEMBOURGISH |
|
TELUGU |
|
CZECH |
|
MACEDONIAN |
|
THAI |
|
DANISH |
|
MALAGASY |
|
TIBETAN |
|
DIVEHI |
|
MALAY |
|
TIGRINYA |
|
DUTCH |
|
MALAYALAM |
|
TONGA |
|
DZONGKHA |
|
MALTESE |
|
TSONGA |
|
ENGLISH |
|
MANX |
|
TSWANA |
|
ESPERANTO |
|
MAORI |
|
TURKISH |
|
ESTONIAN |
|
MARATHI |
|
TURKMEN |
|
EWE |
|
MARSHALLESE |
|
TWI |
|
FAROESE |
|
MOLDAVIAN |
|
UIGHUR |
|
FIJIAN |
|
MONGOLIAN |
|
UKRAINIAN |
|
FINNISH |
|
NAURU |
|
URDU |
|
FRENCH |
|
NAVAJO |
|
UZBEK |
|
FRISIAN |
|
NDONGA |
|
VENDA |
|
FULAH |
|
NEPALI |
|
VIETNAMESE |
|
GALLEGAN |
|
NORTHERN SAMI |
|
VOLAPUK |
|
GANDA |
|
NORTH NDEBELE |
|
WALLOON |
|
GEORGIAN |
|
NORWEGIAN |
|
WELSH |
|
GERMAN |
|
NORWEGIAN BOKMAL |
|
WOLOF |
|
GREEK |
|
NORWEGIAN NYNORSK |
|
XHOSA |
|
GREENLANDIC |
|
NYANJA |
|
YIDDISH |
|
GUARANI |
|
OCCITAN |
|
YORUBA |
|
GUJARATI |
|
OJIBWA |
|
ZHUANG |
|
HAITIAN |
|
ORIYA |
|
ZULU |
|
HAUSA |
|
OROMO |
|
||
HEBREW |
|
OSSETIAN |
|
Country Options
AFGHANISTAN |
|
GIBRALTAR |
|
PALESTINE |
|
ALAND ISLANDS |
|
GREECE |
|
PANAMA |
|
ALBANIA |
|
GREENLAND |
|
PAPUA NEW GUINEA |
|
ALGERIA |
|
GRENADA |
|
PARAGUAY |
|
AMERICAN SAMOA |
|
GUADELOUPE |
|
PERU |
|
ANDORRA |
|
GUAM |
|
PHILIPPINES |
|
ANGOLA |
|
GUATEMALA |
|
PITCAIRN |
|
ANGUILLA |
|
GUERNSEY |
|
POLAND |
|
ANTARCTICA |
|
GUINEA |
|
PORTUGAL |
|
ANTIGUA AND BARBUDA |
|
GUINEA BISSAU |
|
PUERTO RICO |
|
ARGENTINA |
|
GUYANA |
|
QATAR |
|
ARMENIA |
|
HAITI |
|
REUNION |
|
ARUBA |
|
HEARD ISLAND AND MCDONALD ISLANDS |
|
ROMANIA |
|
AUSTRALIA |
|
HONDURAS |
|
RUSSIA |
|
AUSTRIA |
|
HONG KONG |
|
RWANDA |
|
AZERBAIJAN |
|
HUNGARY |
|
SAINT BARTHELEMY |
|
BAHAMAS |
|
ICELAND |
|
SAINT HELENA |
|
BAHRAIN |
|
INDIA |
|
SAINT KITTS AND NEVIS |
|
BANGLADESH |
|
INDONESIA |
|
SAINT LUCIA |
|
BARBADOS |
|
IRAN |
|
SAINT MARTIN |
|
BELARUS |
|
IRAQ |
|
SAINT PIERRE AND MIQUELON |
|
BELGIUM |
|
IRELAND |
|
SAINT VINCENT AND THE GRENADINES |
|
BELIZE |
|
ISLE OF MAN |
|
SAMOA |
|
BENIN |
|
ISRAEL |
|
SAN MARINO |
|
BERMUDA |
|
ITALY |
|
SAO TOME AND PRINCIPE |
|
BHUTAN |
|
JAMAICA |
|
SAUDI ARABIA |
|
BOLIVIA |
|
JAPAN |
|
SENEGAL |
|
BONAIRE SINT EUSTASIUS AND SABA |
|
JERSEY |
|
SERBIA |
|
BOSNIA AND HERZEGOVINA |
|
JORDAN |
|
SERBIA AND MONTENEGRO |
|
BOTSWANA |
|
KAZAKHSTAN |
|
SEYCHELLES |
|
BOUVET ISLAND |
|
KENYA |
|
SIERRA LEONE |
|
BRAZIL |
|
KIRIBATI |
|
SINGAPORE |
|
BRITISH INDIAN OCEAN TERRITORY |
|
KUWAIT |
|
SINT MAARTEN |
|
BRITISH VIRGIN ISLANDS |
|
KYRGYZSTAN |
|
SLOVAKIA |
|
BRUNEI |
|
LAOS |
|
SLOVENIA |
|
BULGARIA |
|
LATVIA |
|
SOLOMON ISLANDS |
|
BURKINA FASO |
|
LEBANON |
|
SOMALIA |
|
BURUNDI |
|
LESOTHO |
|
SOUTH AFRICA |
|
CAMBODIA |
|
LIBERIA |
|
SOUTH GEORGIA AND THE SOUTH SANDWICH ISLANDS |
|
CAMEROON |
|
LIBYA |
|
SOUTH KOREA |
|
CANADA |
|
LIECHTENSTEIN |
|
SOUTH SUDAN |
|
CAPE VERDE |
|
LITHUANIA |
|
SPAIN |
|
CAYMAN ISLANDS |
|
LUXEMBOURG |
|
SRI LANKA |
|
CENTRAL AFRICAN REPUBLIC |
|
MACAO |
|
SUDAN |
|
CHAD |
|
MACEDONIA |
|
SURINAME |
|
CHILE |
|
MADAGASCAR |
|
SVALBARD AND JAN MAYEN |
|
CHINA |
|
MALAWI |
|
SWAZILAND |
|
CHRISTMAS ISLAND |
|
MALAYSIA |
|
SWEDEN |
|
COCOS ISLANDS |
|
MALDIVES |
|
SWITZERLAND |
|
COLOMBIA |
|
MALI |
|
SYRIA |
|
COMOROS |
|
MALTA |
|
TAIWAN |
|
CONGO |
|
MARSHALL ISLANDS |
|
TAJIKISTAN |
|
COOK ISLANDS |
|
MARTINIQUE |
|
TANZANIA |
|
COSTA RICA |
|
MAURITANIA |
|
THAILAND |
|
COTE DIVOIRE |
|
MAURITIUS |
|
THE DEMOCRATIC REPUBLIC OF CONGO |
|
CROATIA |
|
MAYOTTE |
|
TIMOR LESTE |
|
CUBA |
|
MEXICO |
|
TOGO |
|
CURACAO |
|
MICRONESIA |
|
TOKELAU |
|
CYPRUS |
|
MOLDOVA |
|
TONGA |
|
CZECH REPUBLIC |
|
MONACO |
|
TRINIDAD AND TOBAGO |
|
DENMARK |
|
MONGOLIA |
|
TUNISIA |
|
DJIBOUTI |
|
MONTENEGRO |
|
TURKEY |
|
DOMINICA |
|
MONTSERRAT |
|
TURKMENISTAN |
|
DOMINICAN REPUBLIC |
|
MOROCCO |
|
TURKS AND CAICOS ISLANDS |
|
ECUADOR |
|
MOZAMBIQUE |
|
TUVALU |
|
EGYPT |
|
MYANMAR |
|
UGANDA |
|
EL SALVADOR |
|
NAMIBIA |
|
UKRAINE |
|
EQUATORIAL GUINEA |
|
NAURU |
|
UNITED ARAB EMIRATES |
|
ERITREA |
|
NEPAL |
|
UNITED KINGDOM |
|
ESTONIA |
|
NETHERLANDS |
|
UNITED STATES |
|
ETHIOPIA |
|
NETHERLANDS ANTILLES |
|
UNITED STATES MINOR OUTLYING ISLANDS |
|
FALKLAND ISLANDS |
|
NEW CALEDONIA |
|
URUGUAY |
|
FAROE ISLANDS |
|
NEW ZEALAND |
|
US VIRGIN ISLANDS |
|
FIJI |
|
NICARAGUA |
|
UZBEKISTAN |
|
FINLAND |
|
NIGER |
|
VANUATU |
|
FRANCE |
|
NIGERIA |
|
VATICAN |
|
FRENCH GUIANA |
|
NIUE |
|
VENEZUELA |
|
FRENCH POLYNESIA |
|
NORFOLK ISLAND |
|
VIETNAM |
|
FRENCH SOUTHERN TERRITORIES |
|
NORTHERN MARIANA ISLANDS |
|
WALLIS AND FUTUNA |
|
GABON |
|
NORTH KOREA |
|
WESTERN SAHARA |
|
GAMBIA |
|
NORWAY |
|
YEMEN |
|
GEORGIA |
|
OMAN |
|
ZAMBIA |
|
GERMANY |
|
PAKISTAN |
|
ZIMBABWE |
|
GHANA |
|
PALAU |
|
Objects
| Short Id | Name | Short Id | Name |
|---|---|---|---|
|
airplane |
|
apple |
|
backpack |
|
barbecue |
|
bed |
|
bench |
|
bicycle |
|
bird |
|
boat |
|
book |
|
bottle |
|
bowl |
|
bus |
|
cake |
|
can |
|
car |
|
cat |
|
cell phone |
|
chair |
|
cigarette |
|
clock |
|
couch |
|
cow |
|
cup |
|
dining table |
|
dog |
|
donut |
|
elephant |
|
fire hydrant |
|
foreo espada |
|
foreo iris |
|
foreo issa |
|
foreo luna |
|
fork |
|
hair drier |
|
handbag |
|
horse |
|
keyboard |
|
kite |
|
knife |
|
laptop |
|
lipstick |
|
microwave |
|
motorcycle |
|
mouse |
|
oven |
|
parking meter |
|
payment terminal |
|
person |
|
pizza |
|
potted plant |
|
Red Bull can |
|
refrigerator |
|
sandwich |
|
scissors |
|
sink |
|
skateboard |
|
skis |
|
snowboard |
|
spoon |
|
sports ball |
|
stadium scoreboard |
|
stop sign |
|
suitcase |
|
surfboard |
|
teeth |
|
tennis racket |
|
tie |
|
toaster |
|
toilet |
|
toothbrush |
|
traffic light |
|
train |
|
truck |
|
tv |
|
umbrella |
|
vase |
|
wine glass |
Scenes
| Short Id | Name | Short Id | Name |
|---|---|---|---|
|
airport terminal |
|
amusement park |
|
anechoic chamber |
|
athletic field |
|
atrium |
|
auditorium |
|
badminton court |
|
balcony veranda |
|
ballroom |
|
bar |
|
baseball field |
|
basketball court |
|
bathroom |
|
batters box |
|
bazaar indoor |
|
beach |
|
bedroom |
|
boardwalk |
|
boat deck |
|
booth |
|
bowling alley |
|
boxing ring |
|
bridge |
|
building |
|
bus interior |
|
cafeteria |
|
campsite |
|
campus |
|
canal |
|
car interior |
|
casino indoor |
|
castle |
|
cavern indoor |
|
cemetery |
|
chalet |
|
church |
|
church indoor |
|
closet |
|
coast |
|
cockpit |
|
conference center |
|
conference room |
|
construction site |
|
corral |
|
courthouse |
|
courtroom |
|
courtyard |
|
covered bridge |
|
creek |
|
desert |
|
desert sand |
|
diner |
|
discotheque |
|
door |
|
doorway |
|
driveway |
|
engine room |
|
escalator |
|
exhibition |
|
field |
|
fire station |
|
fishpond |
|
football field |
|
forest |
|
forest road |
|
fountain |
|
garage |
|
garbage dump |
|
gas station |
|
general store indoor |
|
golf course |
|
gym |
|
harbor |
|
highway |
|
home office |
|
hospital |
|
house |
|
ice skating rink |
|
igloo |
|
indoor |
|
industrial area |
|
islet |
|
jacuzzi |
|
kitchen |
|
lake |
|
laundromat |
|
library |
|
lido deck |
|
lighthouse |
|
living room |
|
lobby |
|
locker room |
|
market outdoor |
|
marsh |
|
martial arts gym |
|
moat |
|
mosque |
|
mountain |
|
movie theater |
|
nursery |
|
ocean |
|
office |
|
oilrig |
|
open nature |
|
orchard |
|
outdoor |
|
palace |
|
panel table |
|
pantry |
|
park garden |
|
parking indoor |
|
parking outdoor |
|
pavilion |
|
phone booth |
|
plaza |
|
podium |
|
pond |
|
pub |
|
racecourse |
|
raceway |
|
raft |
|
restaurant |
|
riding arena |
|
ring |
|
river |
|
rostrum |
|
ruin |
|
runway |
|
sandbox |
|
sauna |
|
school |
|
server room |
|
shopping mall |
|
skatepark |
|
ski |
|
skyscraper |
|
squash court |
|
stable |
|
stage |
|
staircase |
|
store |
|
store outdoor |
|
street |
|
subway |
|
subway interior |
|
swamp |
|
swimming pool |
|
tennis |
|
theater |
|
tower |
|
train interior |
|
train station |
|
valley |
|
vegetable garden |
|
village |
|
vineyard |
|
volcano |
|
volleyball court |
|
volleyball court indoor |
|
waterfall |
|
wave |
|
wind farm |
|
windmill |
|
wine cellar bottle storage |
|
winter nature |
|
yard |
Family Status
| Short Id | Name | Short Id | Name |
|---|---|---|---|
|
Married |
|
Parents |
|
Senior |
|
Single |
Occupations
| Short Id | Name | Short Id | Name |
|---|---|---|---|
|
Accountant |
|
Actor |
|
Ambulanceman |
|
Architect |
|
Artist/ Art |
|
Author/Writer |
|
Blogger |
|
Celebrity |
|
Comedian |
|
Communication |
|
Construction Worker |
|
Consultant |
|
Customer service |
|
Designer |
|
DJ |
|
Engineer |
|
Entrepreneur |
|
Executive manager |
|
Financial Analyst |
|
Firefighter |
|
Health worker |
|
Hospitality |
|
Human Resources Professional |
|
Investor |
|
IT professional |
|
Journalist |
|
Kitchen staff |
|
Lawyer |
|
Manufacturing |
|
Marketing |
|
Military |
|
Model |
|
Musician |
|
Photographer |
|
Policeman |
|
Promoter |
|
Public Service Worker |
|
Publisher |
|
Realtor |
|
Sales |
|
Scientist |
|
Security officer |
|
Social Media |
|
Aid worker |
|
Sportsperson |
|
Student |
|
Stylist |
|
Teacher |
|
Trainer/ Coach |
|
Transportation |
|
TV/radio host |
Interests
| Short Id | Name | Short Id | Name |
|---|---|---|---|
|
Advertising & Marketing |
|
Animals |
|
Apparel |
|
Art |
|
Automotives General |
|
Boats & Watercraft |
|
Business News |
|
Business Services |
|
Celebrities & Entertainment News |
|
Classic Vehicles |
|
Colleges & Universities |
|
Comics & Animation |
|
Computer Hardware |
|
Consumer Electronics |
|
Crafts |
|
Discount & Outlet Stores |
|
Employment |
|
Face & Body Care |
|
Family and Parenting |
|
Fantasy Sports |
|
Fashion world |
|
Finance |
|
Fitness & Health |
|
Food & Drinks |
|
Fun |
|
Games & Puzzles |
|
Gardening & Landscaping |
|
General beauty |
|
General Education |
|
Global News |
|
Government |
|
Home Furnishings and Improvement |
|
Humor |
|
Legal |
|
Literature/Books |
|
Motorcycles |
|
Movies |
|
Music & Audio |
|
Online/ Video Games |
|
Outdoors |
|
Primary & Secondary Schooling (K 12) |
|
Programming |
|
Real Estate |
|
Science |
|
Social Media |
|
Sports |
|
Toys |
|
Travel |
|
TV |
|
Vehicles General |
|
Weather |
Devices
| Name | Mapped Id | Official website | OFFICIAL_WEBSITE |
|---|---|---|---|
|
MOBILE_ANDROID |
|
MOBILE_IOS |
|
MOBILE_WINDOWS |
|
MOBILE_BLACKBERRY |
|
MOBILE_OTHER |
|
TABLET_ANDROID |
|
TABLET_IOS |
|
TABLET_WINDOWS |
|
TABLET_OTHER |
|
PC |
|
BOT |
|
EXTERNAL_WEBSITE |
API Account
Access Token
Demo
To try the Talkwalker API, you can use the access token demo (access_token=demo). With this token you can try the Search API (results and histogram) and the streaming API.
Accessing the Talkwalker API with this token, will not return any social media results, only results from blogs, forums and news are returned. (this token can be used for testing only)
Your own Access Token
To use the Talkwalker API with the topics from your Talkwalker or to get results from social media you need to apply and get your own access tokens.
-
read_writeaccess tokens are necessary for search, channel monitoring, updating and deleting documents in a project and for creating streams, deleting streams, setting panels and setting rules. -
authenticationaccess tokens are necessary when using the Authentication API.
To get an access token please contact us.
Credits / Pricing
Monthly Reset of Credits
The credits will be reset every month, on the day of the subscription at 03:00 UTC. (Note that the monthly new results in Talkwalker projects are reset on the first of a new month at 0:00 UTC)
Remaining Credits Endpoint
The endpoint https://api.talkwalker.com/api/v1/status/credits is used to get an overview of consumed credits and API calls.
{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v1/status/credits?access_token=demo",
"result_creditinfo" {
"used_credits_monthly" : 0,
"used_credits_onetime" : 0,
"remaining_credits_monthly" : 0,
"remaining_credits_onetime" : 0,
"next_billing_period" : 1419634800000,
"estimate_credits_used_until_end_of_billing_period" : 0,
"monthly_total" : 0
}
}FAQ
How to get all new documents from a Talkwalker project?
The following command accesses data from a Talkwalker project. This data access takes place in real-time data and applies the rules defined in the given Talkwalker project.
curl -XGET 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/result?access_token=<access_token>'Using a POST request instead of a GET request allows for additional filtering of the results, for example by setting a query.
curl -XPOST 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/result?access_token=<access_token>'
-d '{"q":"<query>"}'
-H "Content-Type: application/json; charset=UTF-8"What does an empty collector query mean?
It is possible to leave the collector_query empty when creating a new collector:
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {}}'
-H "Content-Type: application/json; charset=UTF-8"This collector will not collect any live data, instead it only serves as a target for export tasks.
How to get all documents from a Talkwalker project for a certain time period in the past?
In order to access past results from a Talkwalker project, an export task is used, and the exported results are sent to a collector. First, an empty collector is created. Empty collectors do not collect data in real-time, so the newly created collector remains paused while receiving all data from the export task.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {}}'
-H "Content-Type: application/json; charset=UTF-8"Once the collector is created, it is used as target of a project export task. Export tasks require a start date or timestamp and can take an optional stop date or timestamp.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
-d '{"start":"yyyy-MM-dd['T'HH:mm:ss[.SSS][XXX]]", "stop":"yyyy-MM-dd['T'HH:mm:ss[.SSS][XXX]]","target":"<collector_id>"}'
-H "Content-Type: application/json; charset=UTF-8"An example can be found in the Streaming API documentation.
How to get past and live data for my project?
One collector can collect both, past and live data, with a combination of two approaches.
First, a new collector is created for the project in question:
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {"project" : <project_id>}}'
-H "Content-Type: application/json; charset=UTF-8"This collector, from the moment of its creation, collects live data. Thus, the only part missing is the past data.
In order to access the past data, an export task is needed, which is given the newly created collector as target:
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
-d '{"start":"yyyy-MM-dd['T'HH:mm:ss[.SSS][XXX]]", "target":"<collector_id>"}'
-H "Content-Type: application/json; charset=UTF-8"What to do when out of credits during an export task?
If all credits are consumed during an export task, the task is interrupted. All data exported up to that point is sent to the target collector, so the amount of exported data always matches the amount of consumed credits.
How to get an estimation for the number of results of an export task?
There is no means for estimating the number of results beforehand.
However, the cost of the full export task can be estimated by subdividing the complete time period in multiple chunks. Exporting a fraction of the whole first can provide a rough picture of the dimension of the whole. Similarly, one can go chunk by chunk through a larger time frame and adapt the estimation after each finished task.
Imagine an export task that is supposed to export result data for a period of two years:
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
-d '{"start":"2016-09-01", "stop":"2018-09-01","target":"<collector_id>"}'
-H "Content-Type: application/json; charset=UTF-8"This can also be subdivided 24 tasks of 1 month each:
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
-d '{"start":"2016-09-01", "stop":"2016-10-01","target":"<collector_id>"}'
-H "Content-Type: application/json; charset=UTF-8"Still, the resulting estimation is never guaranteed to fully reflect the real outcome. Spikes in the data, for example, can’t be estimated this way.
Is it possible to get results for multiple projects with one collector?
No, this is not possible. In order to receive results for multiple projects, one collector per project is needed.
How to limit the number of results of an export task?
There is currently no way to limit the number of results of an export task.
How to get the results of a stream from a certain point onward?
When creating a collector, a stream can be connected to it. All data found by that stream from this point on is collected by the collector.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>?access_token=<access_token>'
-d '{"rules" : [{"rule_id" : "<some_rule_id>", "query":"<some_query>"}]}'
-H "Content-Type: application/json; charset=UTF-8"curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {"collector_query" : {"streams" : ["<stream_id>"]}}}'
-H "Content-Type: application/json; charset=UTF-8"When displaying the results, there are two types of chunks: CT_RESULT, containing the matching documents, and CT_CONTROL containing control information.
{
"chunk_type": "CT_CONTROL",
"chunk_control": {
"connection_id": "#pch41wmpsxsh#",
"resume_offset": "<resume_token>",
"collector_id": "<collector_id>"
}
}Among others, each CT_CONTROL chunk contains an offset value "resume_offset", which can be used as a request parameter.
The result access will then start at the beginning of the slice with that resume_token and return all results from that point on.
curl -XGET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?access_token=<access_token>&resume_offset=<resume_token>'How to get only certain topics from a project
The following command creates a collector that returns in real time all documents from a list of topics of a Talkwalker project.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {"project_topics" : {"project": <project_id>, "topics":["<topic_id>"]}}}'
-H "Content-Type: application/json; charset=UTF-8"Topics that were removed from the project, but included in the request, are not taken into consideration.
How to get only certain topics from an export task?
Contrary to the above limitation for only some topics of a project in a collector, this is currently not supported by export tasks. Export tasks can only be based on either a complete project, a stream or a query.
How to get the IDs of Talkwalker topics?
To get a list of the topics defined in a Talkwalker project use the project_id and the access_token on the https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/resources endpoint.
Optionally, the filter type can be set if we want to obtain only search-topics: type=search
curl -XGET 'https://api.talkwalker.com/api/v2/talkwalker/p/<project_id>/resources?access_token=<access_token>&type=search'The result, using the above filter, has the form:
{
"status_code" : "0",
"status_message" : "OK",
"request" : "GET /api/v3/talkwalker/p/<project_id>/resources?access_token=<access_token>&type=search",
"result_resources" : {
"projects" : [ {
"id" : "<project_id>",
"title" : "Air France",
"topics" : [ {
"id" : "2p1nevfo_121244b12ade",
"title" : "Category 1",
"nodes" : [ {
"id" : "l9gb1vj7_9utd4cawszq7",
"title" : "topic 1"
}, {
"id" : "g8wf5sd4_8svs0cfghje8",
"title" : "topic 2"
} ]
}, {
"id" : "kj241kj4_h214jhv21l2a",
"title" : "Catergory 2",
"nodes" : [ {
"id" : "w6fc8sf4_4fds6hdgsjd1",
"title" : "topic 1"
} ]
} ]
} ]
}
}To get results for all projects in 'search' use search as topic ID.
To use a single topic, use the ID of the topic (for example w6fc8sf4_4fds6hdgsjd1 for topic 1 of category 2).
How to eliminate comments from a stream?
To remove comments and retrieve only the original documents add -is:comment to the rules of a stream.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>?access_token=<access_token>'
-d '{"rules" : [{"rule_id" : "<rule_id>", "query":"-is:comment"}]}'
-H "Content-Type: application/json; charset=UTF-8"Another possibility consists in using -is:comment as a query parameter when reading results from a stream.
curl -XGET 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/results?access_token=<access_token>&q=-is:comment'How to get past documents of a Talkwalker project that include special keywords
First, an empty collector is created.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>?access_token=<access_token>'
-d '{"collector_query" : {}}'
-H "Content-Type: application/json; charset=UTF-8"This collector is set as target of a project export task.
curl -XPUT 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
-d '{"start":"<date>", "stop":"<date>","target":"<collector_id>", "query":"keyword-1 AND keyword-2"}'
-H "Content-Type: application/json; charset=UTF-8"The above export task sends all those document to the collector which were published in the given timeframe and which match the query. The query in this example requires documents to include two keywords.
How to use a single stream for multiple applications / clients?
In order to use one single stream to retrieve data for more than one application / client, we set one separate rule per application.
curl -XPUT https://api.talkwalker.com/api/v3/stream/s/<stream_id>?access_token=<access_token>
-d '{"rules":[{"rule_id" : "rule-app-1", "query" : "<query>"},{"rule_id" : "rule-app-2", "query" : "<query>"}]}'After creating this stream, we receive in real time those results which match one of both queries.
curl -XGET 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/results?access_token=<access_token>'The returned results are in the format below.
The documents can be separated using the matched field, indicating which rule the result belongs to and thus, which application is concerned.
{
"chunk_type" : "CT_RESULT",
"chunk_result" : {
"data" : {
"data" : { <default result data (see simple search)> },
"highlighted_data" : [ {
"matched" : {
"rule_id" : "rule-app-1"
"stream_id" : "<stream_id>"
}
"title_snippet" : "<title_snippet_for_rule>",
"content_snippet" : "<content_snippet_for_rule>"
} ]
}
}
}How to get the number of results grouped by media types?
The Talkwalker API provides only documents and histograms. To group results into custom sets, you have to get all the results and then compute those sets locally. Alternatively you can perform separate searches (or histograms) for each of the groups you want to create (use the Talkwalker query syntax to restrict the results to those matching a single group).
Code Examples
Streaming Client Examples
PHP
Note: This example needs the php cURL library and PHP 5.5.
<?php
class TalkwalkerApiStreamingV3ClientExample {
const READ_BEFORE_TERMINATE = 3;
private $url;
private $token;
private $streamId;
private $collectorId;
private $resumeOffset;
private $ruleId;
private $finished = false;
private $unprocessedData = '';
private $currentLine = 0;
private $headerSize = -1;
private $header = '';
private $waitForRetry = 0;
private $errorData = '';
public function __construct($url, $token, $streamId, $collectorId, $resumeOffset, $ruleId) {
$this->url = $url;
$this->token = $token;
$this->streamId = $streamId;
$this->collectorId = $collectorId;
$this->resumeOffset = $resumeOffset;
$this->ruleId = $ruleId;
}
public function run() {
$this->deleteStream();
$this->deleteCollector();
$this->createStream();
$this->createCollectorWithStream();
while (!$this->finished) {
$this->unprocessedData = '';
$this->errorData = '';
$this->headerSize = -1;
$this->header = '';
$url = $this->url . '/v3/stream/c/' . $this->collectorId . '/results?access_token='
. $this->token;
if (!empty($this->resumeOffset)) {
$url .= '&resume_offset=' . $this->resumeOffset;
}
$ch = curl_init($url);
$this->setDefaultCurlOptions($ch);
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_WRITEFUNCTION, [$this, "readStream"]);
curl_exec($ch);
$httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if (curl_errno($ch) == 0 && $httpStatus == 200) {
$this->finished = true;
}
// else: error occurred
if ($httpStatus > 0 && $httpStatus != 200) {
$this->onStatusError($this->errorData);
}
curl_close($ch);
if (!$this->finished) {
if ($this->waitForRetry > 0) {
echo "SERVICE UNAVAILABLE \n";
echo "WAITING " . $this->waitForRetry . "s UNTIL RETRYING\n";
sleep($this->waitForRetry);
$this->waitForRetry = 0;
} else {
sleep(60);
}
}
}
$this->deleteCollector();
$this->deleteStream();
}
private function readStream($ch, $data) {
$resultsToRead = TalkwalkerApiStreamingV3ClientExample::READ_BEFORE_TERMINATE;
$httpStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$this->unprocessedData = $this->unprocessedData . $data;
// read the header when it is complete
if ($this->headerSize < $headerSize) {
$this->headerSize = $headerSize;
$headerComplete = false;
} else {
$headerComplete = true;
}
$partialHeader = substr($this->unprocessedData, 0, $headerSize);
if ($headerComplete && $this->header == '') {
$this->header = substr($this->unprocessedData, 0, $headerSize);
$this->unprocessedData = substr($this->unprocessedData, $headerSize);
}
if ($headerComplete && $httpStatus == 200) {
$arrayData = explode("\r\n", $this->unprocessedData);
$count = count($arrayData);
$from = $this->currentLine;
$this->currentLine = $count;
for ($i = $from; $i < $count; $i++) {
$line = $arrayData[$i];
// try parse json
if (strlen($line) > 0) {
$json = json_decode($line);
if ($json == NULL) {
// put it back only if last element
if ($i == $count - 1) {
$this->unprocessedData = $line;
} else {
$this->finished = true;
$this->handleParseError($line);
return 0;
}
} else {
if (isset($json->chunk_type)) {
switch ($json->chunk_type) {
case "CT_ERROR":
$this->handleStreamError($json->chunk_error);
break;
case "CT_CONTROL":
if (isset($json->chunk_control)) {
$this->resumeOffset = $json->chunk_control->resume_offset;
}
$this->handleStreamControl($json->chunk_control);
break;
case "CT_RESULT":
$this->handleStreamResult($json->chunk_result);
$resultsToRead--;
if ($resultsToRead == 0) {
$this->finished = true;
return 0;
}
break;
default:
$this->unhandledStreamChunk($json);
break;
}
} else {
$this->unhandledStreamChunk($json);
break;
}
}
} else {
$this->unprocessedData = '';
}
}
} elseif ($httpStatus == 503) {
$headerArray = $this->parseHeader($partialHeader);
if (array_key_exists('Retry-After', $headerArray)) {
$this->waitForRetry = $headerArray['Retry-After'];
}
} else {
$this->errorData = $this->errorData . $data;
}
return strlen($data);
}
private function createStream() {
echo "CREATING STREAM\n";
$url = $this->url . '/v3/stream/s/' . $this->streamId . '?access_token=' . $this->token;
if ($this->performRequest($url, [
CURLOPT_CUSTOMREQUEST => 'PUT',
CURLOPT_POSTFIELDS => json_encode([
'rules' => [
[
'rule_id' => $this->ruleId,
'query' => 'published:>0'
]
]
])
])) {
echo 'CREATED STREAM: ' . $this->streamId . "\n";
}
}
private function createCollectorWithStream() {
echo "CREATING COLLECTOR\n";
$url = $this->url . '/v3/stream/c/' . $this->collectorId . '?access_token=' . $this->token;
if ($this->performRequest($url, [
CURLOPT_CUSTOMREQUEST => 'PUT',
CURLOPT_POSTFIELDS => json_encode([
'collector_query' => [
'streams' => [$this->streamId]
]
])
])) {
echo 'CREATED COLLECTOR: ' . $this->collectorId . "\n";
}
}
private function deleteStream() {
$url = $this->url . '/v3/stream/s/' . $this->streamId . '?access_token=' . $this->token;
if ($this->performRequest($url, [
CURLOPT_CUSTOMREQUEST => 'DELETE'
])) {
echo 'DELETED STREAM: ' . $this->streamId . "\n";
}
}
private function deleteCollector() {
$url = $this->url . '/v3/stream/c/' . $this->collectorId . '?access_token=' . $this->token;
if ($this->performRequest($url, [
CURLOPT_CUSTOMREQUEST => 'DELETE'
])) {
echo 'DELETED COLLECTOR: ' . $this->collectorId . "\n";
}
}
private function performRequest($url, $curlOptions) {
$ch = curl_init($url);
$this->setDefaultCurlOptions($ch);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt_array($ch, $curlOptions);
$result = curl_exec($ch);
$answer = json_decode($this->getPayload($ch, $result));
curl_close($ch);
if ($answer != null && $answer->status_code != '0') {
echo 'ERROR: ' . $answer->status_message . "\n";
return false;
}
return true;
}
private function getPayload($ch, $data) {
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
return trim(substr($data, $headerSize));
}
private function parseHeader($header) {
$headers = [];
foreach (explode("\r\n", $header) as $i => $line) {
if ($i === 0) {
$headers['http_code'] = $line;
} else {
if ($line != '') {
list ($key, $value) = explode(': ', $line);
$headers[$key] = $value;
}
}
}
return $headers;
}
private function onStatusError($str) {
echo "START ERROR \n{$str}\n";
}
private function handleParseError($str) {
echo "Could not parse '{$str}'\n";
}
private function handleStreamError($err) {
echo "ERROR\n";
print_r($err);
}
private function handleStreamControl($ctrl) {
echo 'CONTROL: ' . json_encode($ctrl) . "\n";
echo 'UPDATED RESUME_OFFSET TO ' . $this->resumeOffset . "\n";
}
private function handleStreamResult($res) {
$url = empty($res->data->data->url)
? 'URL not available. Provider: ' . $res->data->data->external_provider
: $res->data->data->url;
echo "RESULT: $url\n";
}
private function unhandledStreamChunk($json) {
echo "UNHANDLED\n";
print_r($json);
}
private function setDefaultCurlOptions($ch) {
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 90);
curl_setopt($ch, CURLOPT_FAILONERROR, false);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'PhpExampleClient/1.0.0');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Cache-Control: no-cache', 'Pragma: no-cache',
'Content-Language: en-US', 'Content-Type: application/json']);
}
}
echo "START\n";
$options = getopt('u:t:s:c:o:r:');
$url = $options['u'];
$token = $options['t'];
$streamId = $options['s'];
$collectorId = $options['c'];
$resumeOffset = $options['o'];
$ruleId = $options['r'];
$example = new TalkwalkerApiStreamingV3ClientExample($url, $token, $streamId, $collectorId,
$resumeOffset, $ruleId);
$example->run();
echo "DONE\n";Java
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.util.HashMap;
import java.util.Map;
import java.util.zip.GZIPInputStream;
import org.apache.commons.io.IOUtils;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.fasterxml.jackson.databind.node.ObjectNode;
/**
* Example class can be used as an example.
* It is invoked via the ExampleTest class in this test case
*/
public class TalkwalkerApiStreamingV3ClientExample {
private final String url;
private final String token;
private final String streamId;
private final String collectorId;
private String resumeOffset;
private final String ruleId;
private static final int READ_BEFORE_TERMINATE = 3;
TalkwalkerApiStreamingV3ClientExample(String url, String token, String streamId,
String collectorId, String resumeOffset,
String ruleId) {
this.url = url;
this.token = token;
this.streamId = streamId;
this.collectorId = collectorId;
this.resumeOffset = resumeOffset;
this.ruleId = ruleId;
}
public void run() throws InterruptedException, IOException {
try {
delete("/v3/stream/c/", collectorId);
} catch (FileNotFoundException e) {
System.out.println("COLLECTOR " + collectorId + " NOT FOUND, NO DELETION NEEDED");
}
try {
delete("/v3/stream/s/", streamId);
} catch (FileNotFoundException e) {
System.out.println("STREAM " + streamId + " NOT FOUND, NO DELETION NEEDED");
}
createStream();
createCollectorWithStream();
boolean finished = false;
while (!finished) {
try {
String _url = url + "/v3/stream/c/" + collectorId + "/results?access_token=" + token;
_url = resumeOffset == null || resumeOffset.isEmpty() ? _url : _url + "&resume_offset="
+ resumeOffset;
URLConnection connection = createConnection(_url);
HttpURLConnection httpConnection = (HttpURLConnection) connection;
httpConnection.setRequestMethod("GET");
httpConnection.setRequestProperty("User-Agent", "JavaExampleClient/1.0.0");
httpConnection.setRequestProperty("Accept-Encoding", "gzip");
httpConnection.connect();
int httpCode = httpConnection.getResponseCode();
// getting the correct input stream
if (httpCode == 200) {
try (InputStream is = httpConnection.getInputStream()) {
try {
finished = readStream(httpConnection, is);
} catch (IOException ioe) {
//stream or connection was interrupted, retry with next iteration
}
}
} else if (httpCode == 503) {
// the service is currently unavailable
int secondsToWait = httpConnection.getHeaderFieldInt("Retry-After", 60);
System.out.println("TEMPORARILY UNAVAILABLE");
System.out.println("WAITING " + secondsToWait + "s UNTIL RETRYING");
Thread.sleep(secondsToWait * 1000);
} else {
// when encountering an error, we exit loop
try (InputStream is = httpConnection.getErrorStream()) {
readError(httpConnection, is);
} catch (IOException e) {
e.printStackTrace();
} finally {
finished = true;
}
}
} catch (IOException ex) {
// try again
ex.printStackTrace();
// sleep a minute
Thread.sleep(60 * 1000);
}
}
delete("/v3/stream/c/", collectorId);
delete("/v3/stream/s/", streamId);
}
private void readError(HttpURLConnection httpConnection, InputStream errorInputStream)
throws IOException {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] dataBuf = new byte[1024 * 1024];
// read answer
while (true) {
int read = errorInputStream.read(dataBuf, 0, dataBuf.length);
if (read == -1) {
break;
}
bos.write(dataBuf, 0, read);
}
InputStream is = new ByteArrayInputStream(bos.toByteArray());
if ((httpConnection.getContentEncoding() != null) && (httpConnection.getContentEncoding()
.equals("gzip"))) {
is = new GZIPInputStream(is);
}
// read json using jackson json (another library may be used here)
JsonFactory factory = new JsonFactory();
ObjectMapper mapper = new ObjectMapper(factory);
TypeReference<HashMap<String, Object>> typeRef = new TypeReference<HashMap<String, Object>>() {
};
mapper.readValue(is, typeRef);
}
private boolean readStream(HttpURLConnection httpConnection, InputStream inputStream)
throws IOException {
// reading the stream and invoking the listener
InputStream is = inputStream;
if ((httpConnection.getContentEncoding() != null) && (httpConnection.getContentEncoding()
.equals("gzip"))) {
is = new GZIPInputStream(is);
}
int resultsToRead = READ_BEFORE_TERMINATE;
BufferedReader reader = new BufferedReader(new InputStreamReader(is, "UTF-8"),
100);
String line;
while ((line = reader.readLine()) != null) {
if (line.isEmpty()) {
continue;
}
// parse json (use an available json parser)
JsonFactory factory = new JsonFactory();
ObjectMapper mapper = new ObjectMapper(factory);
TypeReference<HashMap<String, Object>> typeRef =
new TypeReference<HashMap<String, Object>>() {
};
HashMap<String, Object> o = mapper.readValue(line, typeRef);
Object oType = o.get("chunk_type");
if (oType != null && oType instanceof String) {
String type = (String) oType;
switch (type) {
case "CT_ERROR":
Map<String, Object> errorChunk = getAsMap(o, "chunk_error");
handleStreamError(errorChunk);
break;
case "CT_CONTROL":
Map<String, Object> controlChunk = getAsMap(o, "chunk_control");
if (controlChunk != null) {
resumeOffset = getAsT(controlChunk, "resume_offset", String.class);
}
handleStreamControl(controlChunk);
break;
case "CT_RESULT":
Map<String, Object> resultChunk = getAsMap(o, "chunk_result");
handleStreamResult(resultChunk);
resultsToRead--;
if (resultsToRead == 0) {
return true;
}
break;
default:
unhandledStreamChunk(o);
break;
}
} else {
unhandledStreamChunk(o);
}
}
return false;
}
static Map<String, Object> getAsMap(Map<String, Object> o, String key) {
if (o != null) {
Object oRet = o.get(key);
if (oRet != null && oRet instanceof Map) {
return (Map<String, Object>) oRet;
}
}
return null;
}
static <T> T getAsT(Map<String, Object> o, String key, Class<T> clz) {
if (o != null) {
Object oRet = o.get(key);
if (oRet != null && clz.isInstance(oRet)) {
return (T) oRet;
}
}
return null;
}
private void handleStreamError(Map<String, Object> errorChunk) {
System.out.println("ERROR: " + errorChunk);
}
protected void handleStreamControl(Map<String, Object> controlChunk) {
System.out.println("CONTROL: " + controlChunk);
System.out.println("UPDATED RESUME_OFFSET TO " + resumeOffset);
}
protected void handleStreamResult(Map<String, Object> resultChunk) {
Map<String, Object> resultData = getAsMap(resultChunk, "data");
Map<String, Object> entryData = getAsMap(resultData, "data");
String url = getAsT(entryData, "url", String.class) == null
? "URL not available. Provider: " + getAsT(entryData, "external_provider", String.class)
: getAsT(entryData, "url", String.class);
System.out.println("RESULT: " + url);
}
private void unhandledStreamChunk(Map<String, Object> unhandledChunk) {
System.out.println("UNHANDLED: " + unhandledChunk);
}
private void createStream() throws IOException {
System.out.println("CREATING " + streamId);
String _url = url + "/v3/stream/s/" + streamId + "?access_token=" + token;
URLConnection connection = createConnection(_url);
HttpURLConnection httpConnection = (HttpURLConnection) connection;
httpConnection.setRequestMethod("PUT");
httpConnection.setRequestProperty("User-Agent", "JavaExampleClient/1.0.0");
httpConnection.setRequestProperty("charset", "utf-8");
httpConnection.setDoOutput(true);
httpConnection.setDoInput(true);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
JsonNodeFactory factory = JsonNodeFactory.instance;
ObjectNode arrayEntry = factory.objectNode().put("rule_id", ruleId)
.put("query", "published:>0");
ObjectNode on = factory.objectNode();
on.putArray("rules").add(arrayEntry);
System.out.println(on.toString());
wr.writeBytes(on.toString());
wr.flush();
wr.close();
httpConnection.connect();
int httpCode = httpConnection.getResponseCode();
if (httpCode == 200) {
System.out.println("CREATED " + streamId);
} else {
System.out.println("ERROR");
System.out.println(IOUtils.toString(httpConnection.getInputStream(), "UTF-8"));
}
}
protected void createCollectorWithStream() throws IOException {
System.out.println("CREATING " + collectorId);
String _url = url + "/v3/stream/c/" + collectorId + "?access_token=" + token;
URLConnection connection = createConnection(_url);
HttpURLConnection httpConnection = (HttpURLConnection) connection;
httpConnection.setRequestMethod("PUT");
httpConnection.setRequestProperty("User-Agent", "JavaExampleClient/1.0.0");
httpConnection.setRequestProperty("charset", "utf-8");
httpConnection.setDoOutput(true);
httpConnection.setDoInput(true);
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
JsonNodeFactory factory = JsonNodeFactory.instance;
ObjectNode on = factory.objectNode();
on.putObject("collector_query").putArray("streams").add(streamId);
System.out.println(on.toString());
wr.writeBytes(on.toString());
wr.flush();
wr.close();
httpConnection.connect();
int httpCode = httpConnection.getResponseCode();
if (httpCode == 200) {
System.out.println("CREATED " + collectorId);
} else {
System.out.println("ERROR");
System.out.println(IOUtils.toString(httpConnection.getInputStream(), "UTF-8"));
}
}
private void delete(String path, String id) throws IOException {
String _url = url + path + id + "?access_token=" + token;
URLConnection connection = createConnection(_url);
HttpURLConnection httpConnection = (HttpURLConnection) connection;
httpConnection.setRequestMethod("DELETE");
httpConnection.setRequestProperty("User-Agent", "JavaExampleClient/1.0.0");
httpConnection.setRequestProperty("charset", "utf-8");
httpConnection.setDoOutput(true);
httpConnection.setDoInput(true);
httpConnection.connect();
int httpCode = httpConnection.getResponseCode();
if (httpCode == 200) {
System.out.println("DELETED " + id);
} else {
System.out.println(IOUtils.toString(httpConnection.getInputStream(), "UTF-8"));
}
}
private URLConnection createConnection(String url) throws IOException {
URL request = new URL(url);
URLConnection connection = request.openConnection();
connection.setConnectTimeout(30000);
connection.setReadTimeout(90000);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Language", "en-US");
return connection;
}
}Throubleshooting
Error Codes
| http code | status code | message | description |
|---|---|---|---|
|
|
OK |
Default answer |
|
|
Internal Server Error |
An unexpected exception was encountered. |
|
|
Search Execution Exception |
An unexpected exception was encountered. Related to the search |
|
|
Parameter Missing |
Required parameters are missing. The missing parameters are provided in key 'params'. |
|
|
Error in query |
Could not parse query. The details can be found under 'details'. |
|
|
Invalid parameter value |
A parameter has an unacceptable value. The parameter is listed under 'param' and the details under 'details'. |
|
|
Invalid, missing or inactive access token |
The access token is either missing or the provided value is invalid. |
|
|
Call limit exceeded for this endpoint |
The called endpoint has a limited call frequency, the values should be cached by the client. |
|
|
No credits left. |
The account ran out of credits. |
|
|
API application is inactive |
The API account is inactive. 'appId' gives the ID of that account. |
|
|
No such application linked |
The provided ID is not linked in the API to any project or application. |
|
|
Linked application inactive or deleted |
The linked application is inactive or deleted. |
|
|
Access denied: Insufficient access rights. |
The used access token does not have enough access rights. 'rights_req' will list the required access rights, 'rights_got' lists the access rights provided by that access token. |
|
|
Wrong stream id. No such stream defined. |
A non existing stream was accessed. |
|
|
Invalid operation on document |
The search document modification operation is not supported. 'reason' and 'details' will provide more information. |
|
|
Could not parse json |
The JSON that was passed via POST could not be properly interpreted (it was not in the expected format). |
|
|
Invalid operation on stream |
Modifying a stream failed. See 'reason' for details. |
|
|
Number of rules to set exceeds maximum number of rules |
Exceeded the maximum allowed rules for this API account. 'number_max' is the limit, 'number_available' how many we can save and 'number_saving' the number we tried to save |
|
|
Cannot create any more streams |
Exceeded maximum amount of streams ('number_max') |
|
|
A stream with this name already exists |
The stream 'stream_id' is already defined. |
|
|
Number of sources to set exceeds maximum number of sources |
Exceeded the maximum allowed sources (whitelist or blacklist) for this API account. 'number_max' is the limit, 'number_available' how many we can save and 'number_saving' the number we tried to save. |
|
|
Stream has no rules defined |
Exception when trying to stream with a stream that has no rule defined. |
|
|
Stream got disconnected because newer stream running |
A new stream (same 'stream_id') is connected, so the old stream will be disconnected. |
|
|
Stream got disconnected |
The stream was disconnected due to the given reason. |
|
|
Endpoint or action not found |
The called endpoint was not found. |
|
|
Connection is not secure, must use HTTPS |
Authentication API endpoints need to be called using HTTPS. |
|
|
User was not found in this application |
This user ID does not exist or is not linked to this project. |
|
|
Access to this project is forbidden |
This project can not be accessed with the given access_token. |
|
|
Limit of maximum concurrent streams reached |
Too many streams running in parallel for this account. |
|
|
Could not find rule with id |
A rule with the given ID could not be found. |
|
|
Could not find panel with id |
A panel with the given ID could not be found. |
|
|
Panel is still referenced |
This panel could not be deleted, it is still used in a stream. |
|
|
HTTP Version Not Supported |
The Talkwalker Streaming API supports HTTP 1.1 or newer. |
|
|
Url is malformed |
The given URL for channel monitoring is malformed |
|
|
Could not execute action in Talkwalker |
Error in connecting to a Talkwalker project |
|
|
Access prohibited |
Access prohibited due to access restriction settings |
|
|
Cannot create any more panels |
The maximum number of panels is reached |
|
|
Cannot find a project with this id |
The project with this ID could not be found or is not accessible |
|
|
Request entity too large |
Request entity too large |
|
|
Global search is disabled for this account |
Global search is disabled for this account. |
|
|
Some or all requests of this bulk request failed |
Some or all requests of this bulk request failed |
|
|
Request entity too large |
The sent PUT or POST request is too large (maximum 5120kb) |
|
|
Service Temporarily Unavailable |
The Talkwalker servers are restarting due too an update |
Error Handling
Streaming API
Resuming an interrupted stream
A stream collector can be interrupted for several reason: given maximum of hits (max_hits) reached, end_behaviour configured accordingly, no credits left, server issues or connection problems.
To resume a disconnected stream, set the parameter resume_offset to the resume_offset of the last CT_CONTROL chunk.
This allows to continue streaming from the exact position of that resume_offset on.
{
"chunk_type": "CT_CONTROL",
"chunk_control": {
"connection_id": "<some_connection_id>",
"resume_offset": "<resume_offset>",
"collector_id": "<some_collector_id>"
}
}
curl -XGET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?access_token=<access_token>&resume_offset=<resume_offset>'Document API
Document import fails with "Does not match any xyz" Error
Documents that do not match a projects queries can not be imported into a project. The details of the the error message explain what part of the project was not matched. The provided document must match the settings of the project (languages, countries, source types and blocked sources) and the query of at least one topic.
When importing documents from a specific domain, an extra topic similar to domainurl:"http://my-site.com/" can help to match all uploaded documents.