Collect Data¶
The test and staging packages of the SDK can send what they capture to DEA's collection service, where it trains the models that score sessions. This page shows how a test or staging app attaches the collection transporter's stream delivery: the page's input goes to DEA in pieces while it runs, and the transporter cuts it into sessions on its own, so your app makes no call per screen or per round.
Warning: Only the test and staging packages collect, and they send the raw input on the page — pointer positions, scrolls, which class of key was pressed, device motion — to DEA. Use them only in an app DEA asked you to collect with, with your users' agreement, and never release one. The production package ignores the transporter.
When to use it¶
- A data-collection or QA app that DEA asked you to build
- A staging build of your app whose testers' sessions should reach DEA's training data
Prerequisites¶
- An
appIdandappKeyfor the app — see Accounts and keys - The page is served over HTTPS, as in the Quick Start
- Your Content Security Policy allows
https://poh-api.dep-protocol.cominconnect-src— see Content Security Policy - Your users agreed to collection — see User Consent
Usage¶
Install the packages¶
A staging app installs the staging SDK and the collection transporter. The transporter is published in the test channel only, so it comes from there, at the same release as the SDK:
npm install https://sdk.dep-protocol.com/0.26.0/staging/web/sdk/dea-sdk-0.26.0.tgz \
https://sdk.dep-protocol.com/0.26.0/test/web/ai-backend-transporter/dea-ai-backend-transporter-0.26.0.tgz
A test app installs the test SDK instead: the same command with /test/ in place of /staging/.
The test package also writes a DEBUG entry for every captured object to the console.
| Package | Collects |
|---|---|
Production, /web/sdk/ |
No: the transporter is ignored, and nothing captured leaves the device |
Staging, /staging/web/sdk/ |
Yes |
Test, /test/web/sdk/ |
Yes |
The package decides, not the appId: the same appId collects on a staging package and does not on
the production one.
Attach the stream delivery¶
import { init } from '@dea/sdk';
import { aiBackendTransport } from '@dea/ai-backend-transporter';
const dea = init({
appId: 'YOUR_APP_ID',
appKey: 'YOUR_APP_KEY',
// Nothing is captured, scored or sent until the user agrees.
requireConsent: true,
integrations: [
aiBackendTransport({
kind: 'human',
subject_id: 'tester-0142',
task: 'staging-ai-data',
delivery: 'stream',
}),
],
});
// When the user agrees:
dea.setConsent(true);
That is the whole integration. Where the stream goes and the key it authenticates with are built into the transporter package; there is nothing to configure.
aiBackendTransport throws when the label is not one DEA accepts: a kind other than 'human'
or 'bot', a bot without bot_family, an unknown bot_injection or delivery. Build it where an
exception reaches you, before init.
Label the sessions¶
Every streamed session is filed under the label given to aiBackendTransport.
| Field | Required | Description |
|---|---|---|
kind |
Yes | 'human' for a person, 'bot' for an automated client |
subject_id |
No | Who produced the input: a pseudonymous id of the person or the bot, never personal data. Sessions with the same id can be grouped |
task |
No | What the sessions are for. 'staging-ai-data' when not given |
bot_family |
When kind is 'bot' |
Which bot produced the input |
bot_injection |
No | How a bot's input reached the page: 'framework', 'os-level' or 'human-relay' |
device_model |
No | The handset a person named. It fills device.model only where the browser names none, which is every browser on iOS |
runtime_config |
No | Your own metadata for the sessions, never personal data |
delivery |
Yes, 'stream' |
Without it the transporter uploads each session whole when it ends, which is for DEA's own test apps |
The label is read once, when the SDK starts. To file what follows under another person, dispose of the SDK and start it again with the new label:
import { DeaSdk, init } from '@dea/sdk';
import { aiBackendTransport } from '@dea/ai-backend-transporter';
function startFor(subjectId: string) {
DeaSdk.current?.dispose();
return init({
appId: 'YOUR_APP_ID',
appKey: 'YOUR_APP_KEY',
requireConsent: true,
integrations: [aiBackendTransport({ kind: 'human', subject_id: subjectId, delivery: 'stream' })],
});
}
The running streamed session then ends when its event flow stops: within a second on a desktop, or on a phone or tablet that granted motion; otherwise when the page goes to the background or closes.
Keep the event flow going¶
The transporter ends a streamed session where the page's event flow is interrupted — see Where a streamed session ends. Your app does not need to do anything for a session to end, and it should avoid ending one by accident:
- Do not call
reset()orpause()for a screen, a game or a round.reset()starts a new session, and a pause longer than a second stops the event flow. Callreset()only where a session really ends, such as a logout — see Sessions. - Do not open
alert(),confirm(),prompt()orprint()while collecting. A browser dialog stops the page for as long as it is open, and one open for more than a second ends the session. Use a dialog drawn in the page instead. - Change screens inside the page. A route change in a single-page app (
history.pushState,replaceState, a hash change) keeps the session; a reload or a link to another document ends it.
Examples¶
A staging build that streams while testers play¶
import { DeaSdk, init } from '@dea/sdk';
import { aiBackendTransport } from '@dea/ai-backend-transporter';
// The tester types who they are once, when the app opens.
export function startCollecting(subjectId: string): DeaSdk {
DeaSdk.current?.dispose();
return init({
appId: 'YOUR_APP_ID',
appKey: 'YOUR_APP_KEY',
requireConsent: true,
integrations: [
aiBackendTransport({ kind: 'human', subject_id: subjectId, task: 'staging-ai-data', delivery: 'stream' }),
],
});
}
// Your consent dialog calls this with the answer.
export function answerConsent(agreed: boolean): void {
DeaSdk.current?.setConsent(agreed);
}
// Mark where each game starts and ends: the marks travel with the stream, and do not cut it.
export function gameStarted(game: string): void {
DeaSdk.current?.captureEvent('session_start', { task: game });
}
export function gameEnded(game: string, completed: boolean): void {
DeaSdk.current?.captureEvent('session_end', { task: game, completed });
}
Check that it works¶
- Open the browser's developer tools on the Network tab and use the page for ten seconds.
- You see
POST https://poh-api.dep-protocol.com/api/collect-data/data-training-streamingrequests, gzipped, whose JSON body is{ "data": …, "event_type": … }:session_stream_ingestevery 10 seconds while the page is in front, sooner when a lot is capturedsession_stream_endwhen the session ends, for example when you switch to another tab
data.session_idnames the streamed session. It is not the iddea.getSessionInfo().sessionreturns: one SDK session can stream as several sessions.
The staging package also logs every request, as an http.response entry whose url.path is
/api/collect-data/data-training-streaming.
If nothing is sent:
| Check | Why |
|---|---|
The SDK URL has /staging/ or /test/ in it |
The production package ignores the transporter |
setConsent(true) was called |
With requireConsent, nothing reaches the transporter before it |
| The session lasted 3 seconds or more, with a tap, click, scroll or key press | A session that ends before its first piece is sent, shorter than that or without input, is not sent |
connect-src allows https://poh-api.dep-protocol.com |
The browser blocks the request otherwise |
API¶
| Member | Returns | Description |
|---|---|---|
aiBackendTransport(config) |
Integration |
The collection transporter, for init({ integrations }). With delivery: 'stream', the stream delivery |
Delivery.Stream |
'stream' |
The value of delivery for the stream delivery |
Related¶
- Collecting data — what is sent, when, and where a session ends
- User Consent
- Sessions
- Accounts and keys