EVTC Upload API

API Based on wingman uploadEVTC

Accepts POST /evtc multipart uploads (account, filesize, triggerID, file), runs them through the GW2 Elite Insights parser and lets clients poll the result.

Endpoints

Method	Path	Description
POST	/evtc	Upload a log. Returns { "result": true, "ticket": N }.
GET	/status	Queue size, dequeued count, backlog size and worker count.
GET	/status/:ticket	Position and outcome of a single upload.
GET	/health	Liveness probe.

Observability

Three signals, each where it belongs: metrics in Prometheus, traces in Tempo, logs in Loki, correlated by trace_id.

docker compose up -d brings all three up alongside eiapi. Grafana is expected to run outside the stack, so their query ports are published on the host:

datasource	URL for Grafana	notes
Prometheus	http://localhost:9090	scrapes eiapi:9464 every 15s, keeps 30d
Loki	http://localhost:3100	14d retention, usage reporting off
Tempo	http://localhost:3200	14d retention, receives OTLP/gRPC on 4317, metrics generator on for Traces Drilldown

They bind to 127.0.0.1 because none of them authenticate. If Grafana runs on another machine, change the bind address in docker-compose.yml and put something in front of them. The host side of those port mappings is the first thing to check on a new host: 3100, 3200 and 9090 are popular, and grafana-agent in particular already sits on 9090/9091. A conflict only shows up when a container is actually created, so docker compose restart will not reveal it. Use up -d --force-recreate.

Tempo runs with the metrics generator and the local-blocks processor enabled. Grafana's Traces Drilldown does not only fetch traces, it runs TraceQL metrics queries ({...} | rate() by(resource.service.name)); without the generator those fail with error finding generators: empty ring.

To jump from a log line to its trace, add a derived field to the Loki datasource: regex "trace_id":"(\w+)", name trace_id, internal link to the Tempo datasource with ${__value.raw} as the query.

Metrics

GET /metrics on EI_METRICS_PORT (9464), a separate listener from the API. It scans the retry and dead directories on every scrape, which should not be something the internet can trigger.

metric	type	labels
eiapi_uploads_total	counter	result (accepted/rejected), reason
eiapi_jobs_total	counter	outcome (uploaded/skipped/deferred/failed)
eiapi_wingman_refusals_total	counter	kind, reason
eiapi_parser_failures_total	counter	reason
eiapi_requeued_total	counter	-
eiapi_parser_duration_seconds	histogram	-
eiapi_queue_wait_seconds	histogram	-
eiapi_job_duration_seconds	histogram	-
eiapi_upload_bytes	histogram	-
eiapi_queue_size, eiapi_in_flight, eiapi_workers	gauge	-
eiapi_backlog, eiapi_dead	gauge	-
eiapi_tickets, eiapi_seen, eiapi_dequeued_total	gauge	-

The gauges are filled in at scrape time, so they are as fresh as the scrape interval rather than the 2 minute queue_stats tick. Alerting on the backlog becomes an ordinary rule:

eiapi_backlog > 0
increase(eiapi_wingman_refusals_total{kind="transient"}[10m]) > 0
histogram_quantile(0.95, sum by (le) (rate(eiapi_parser_duration_seconds_bucket[15m])))
sum by (outcome) (rate(eiapi_jobs_total[5m]))

Traces

With EI_OTLP_ENDPOINT set, every upload becomes one trace: an upload span for the request with a child job span covering the queue wait, the parser run and the wingman classification, with the log events attached to the spans. The job span continues the upload's trace even though it runs in another task minutes later, so a queue backlog is visible as a gap in the waterfall rather than as two unrelated traces.

Logs written inside those spans carry trace_id, so a Loki derived field can link straight to Tempo:

{application="ei-runner"} | json | trace_id="7822527d568864e5c6ef499c5e82a89d"

Configure the derived field in the Loki datasource with regex "trace_id":"(\w+)" pointing at the Tempo datasource.

Requeued logs (parked while wingman was down, or recovered after a restart) start a new trace: the original request is long gone by then. They are still tied together by ticket and name in the logs.

Logs

tracing-loki pushes every event to Loki as a JSON line, with application and level as the only labels. Everything else is a field inside the line, so | json gives them all without blowing up label cardinality.

Every event carries a stable event field so nothing has to match on prose:

event	level	fields
upload_received	debug	filesize, bytes
upload_queued	info	ticket, bytes, position
upload_rejected	info	reason (duplicate, name_taken, no_filename, invalid_filename, invalid_account, name_too_long), status
parser_finished	info	parser_elapsed_ms, parsed, status
parser_failed	error	status, stderr
job_finished	info	outcome (uploaded, skipped, deferred, failed), queue_wait_ms, processing_ms
job_failed	error	reason (not_parsed, no_result, parser_timeout, parser_error, upload_failed, unreadable_result)
wingman_refused	warn	kind (transient/permanent), reason (server_busy, unreachable, request_failed, rejected, anonymised, …), response
wingman_recovered	info	requeued
queue_stats	info, every 120s	queue_size, in_flight, workers, backlog, dead, dequeued, tickets, seen

Following a single upload

Uploads run inside a span, and span fields are flattened into every event emitted underneath, so one filter gives the full pipeline of one log, including the lines the worker writes minutes later, and the ones written after it was parked and requeued:

{application="ei-runner"} | json | ticket="42"

The upload span adds account, trigger_id and name (the client filename); the job span adds ticket, name (the stored filename) and attempt. So a log can also be followed without knowing its ticket:

{application="ei-runner"} | json | account="Foo.1234"
{application="ei-runner"} | json | name=~".*20260417-171254.*"

A typical trace for a log that hit a full wingman and went through later:

upload_queued     ticket=2 account=Obs.1234 name=bad.zevtc position=1
parser_finished   ticket=2 parser_elapsed_ms=1 parsed=true
wingman_refused   ticket=2 kind=transient reason=server_busy response=ERR_SERVER_CAPACITY_CRITICAL
job_finished      ticket=2 outcome=deferred queue_wait_ms=0 processing_ms=1 attempt=1
wingman_recovered ticket=3 requeued=1          # a later upload succeeded
job_finished      ticket=2 outcome=uploaded attempt=2

Alerts

# wingman refuses for a reason that is not the log's fault (down, or full)
sum(count_over_time({application="ei-runner"} | json | event="wingman_refused" | kind="transient" [10m])) > 0

# specifically out of capacity
sum(count_over_time({application="ei-runner"} | json | response="ERR_SERVER_CAPACITY_CRITICAL" [10m])) > 0

# backlog is not draining: wingman has been unavailable for a while
max(max_over_time({application="ei-runner"} | json event="event", v="backlog" | event="queue_stats" | unwrap v [30m])) > 0

# logs being given up on
max(max_over_time({application="ei-runner"} | json event="event", v="dead" | event="queue_stats" | unwrap v [1h])) > 0

# the parser itself is broken (crash, timeout)
sum(count_over_time({application="ei-runner"} | json | event=~"parser_failed|job_failed" [15m])) > 3

# queue saturated: work waiting with every parser busy
min(min_over_time({application="ei-runner"} | json event="event", v="queue_size" | event="queue_stats" | unwrap v [15m])) > 10

Dashboard panels

# throughput by outcome
sum by (outcome) (count_over_time({application="ei-runner"} | json | event="job_finished" [5m]))

# parse duration p50/p95
quantile_over_time(0.95, {application="ei-runner"} | json event="event", v="parser_elapsed_ms" | event="parser_finished" | unwrap v [1h])

# how long uploads wait for a free parser
quantile_over_time(0.95, {application="ei-runner"} | json event="event", v="queue_wait_ms" | event="job_finished" | unwrap v [1h])

# queue, in flight and backlog over time
max(max_over_time({application="ei-runner"} | json event="event", v="queue_size" | event="queue_stats" | unwrap v [5m]))
max(max_over_time({application="ei-runner"} | json event="event", v="in_flight" | event="queue_stats" | unwrap v [5m]))
max(max_over_time({application="ei-runner"} | json event="event", v="backlog" | event="queue_stats" | unwrap v [5m]))

# rejected uploads by reason
sum by (reason) (count_over_time({application="ei-runner"} | json | event="upload_rejected" [15m]))

A bare | json turns every field of the line into a label, so an unwrap query returns one series per distinct label set and a stat panel shows a wall of numbers instead of one. Extract only what the query needs, | json event="event", v="backlog", and the unwrapped label is dropped from the result, which leaves a single series. Aggregating queries (sum by (outcome)) are unaffected because the aggregation already discards the other labels.

docs/grafana-dashboard.json is an importable dashboard. It has no datasource variables: the panels point at fixed UIDs prometheus-eiapi, loki-eiapi and tempo-eiapi, so name the datasources that way when adding them. Metrics panels come from Prometheus, the two log panels from Loki and the trace list from Tempo.

Histogram buckets differ per metric, and their resolution is placed where a busy evening actually puts the data: parse duration around 10-60s with a tail to the 15 minute parser timeout, queue wait around 5-30 minutes with room to four hours, upload size in bytes to the body limit. A shared bucket set would put every upload size in +Inf and make the queue wait quantiles unusable exactly when the queue is deep.

Sizing note: with N parsers and a mean parse of P seconds the server can absorb N / P uploads per second. Queue wait grows sharply once arrivals get close to that; at 6 parsers, a 17s mean parse and 0.32 uploads/s, utilisation is 91% and waits of 10-25 minutes are the expected result rather than a fault. Raising EI_PARSER_JOBS is the lever.

Loki being unreachable does not affect the server: tracing-loki retries with backoff and logs its own failures to stdout, so the container log still has everything.

Docker

The image contains eiapi, the .NET 8 runtime and an Elite Insights CLI that installs and updates itself. The parser is not baked into the image.

cp .env.example .env   # optional, only needed to change ports or limits
docker compose up -d

Every published port and the container limits come from .env, with the values in .env.example as the fallbacks, so nothing has to be edited in docker-compose.yml on a host where something already owns 9090. .env is gitignored, so local ports survive a git pull.

On start the entrypoint asks the GitHub API for the latest release of GW2-Elite-Insights-Parser, unpacks GW2EICLI.zip into /opt/ei/versions/<tag> and points /opt/ei/current at it. The check repeats every EI_UPDATE_INTERVAL seconds (5min by default, 0 disables it) and recurring checks stay silent unless something changes.

The check reads the tag that github.com/<repo>/releases/latest redirects to rather than calling api.github.com. The API allows 60 requests per hour per IP for anonymous callers, shared with everything else on that address, and conditional requests do not help, because GitHub counts a 304 Not Modified against the limit too. The redirect costs nothing from that budget.

The release ships the CLI as a framework dependent dll with a Windows only apphost, so it runs as dotnet GuildWars2EliteInsights-CLI.dll; eiapi does that automatically when EI_PARSER_BIN ends in .dll.

Updating is a symlink swap and the previous version is kept, so a parser that is running when the switch happens keeps working and the next log picks up the new version. If GitHub is unreachable the existing installation keeps being used; the container only refuses to start when there is no installation at all yet.

Two volumes: /data for the queue and /opt/ei so a recreated container does not have to download the parser again. A bind mount for /data has to be writable by uid 10001, which is the unprivileged user the container runs as.

The parser sizes its own memory limit from what it can see, which inside a container is the container limit, so give it one (mem_limit in the compose file). EI_PARSER_JOBS parsers run at the same time, each one a full Elite Insights run.

The bundled settings.conf is copied to /etc/eiapi/settings.conf; mount your own over it to override. It must keep SaveAtOut=true, SaveOutTrace=true and UploadToWingman=true. See the comments in the file for why.

Configuration

All configuration comes from the environment. The two parser paths are required; startup fails with a clear message if they are missing or do not exist.

Variable	Required	Default	Description
EI_PARSER_BIN	yes	-	GuildWars2EliteInsights-CLI.dll (run via dotnet) or a native binary.
EI_DOTNET	no	dotnet	Used when EI_PARSER_BIN is a dll.
EI_PARSER_SETTINGS	yes	-	settings.conf handed to the parser via -c.
EI_PARSER_DIR	no	dir of EI_PARSER_BIN	Working directory for the parser process.
EI_PARSER_TIMEOUT_SECS	no	900	Kill a parser run after this many seconds.
EI_PARSER_JOBS	no	CPUs, max 4	Parser processes running at the same time.
EI_DATA_DIR	no	data	Root for uploads/, retry/ and timeouts/. Must be persistent.
EI_BIND_ADDR	no	0.0.0.0	Listen address. Run behind a reverse proxy.
PORT	no	3334	Listen port.
EI_BODY_LIMIT	no	262144000	Max request body size in bytes (250MiB).
EI_RETRY_MAX_ATTEMPTS	no	5	Give up on a repeatedly refused log and move it to dead/.
EI_LOKI_URL	no	unset	Loki endpoint. Logging stays on stdout when unset.
EI_OTLP_ENDPOINT	no	unset	OTLP/gRPC collector for traces, e.g. http://tempo:4317.
EI_SERVICE_NAME	no	eiapi	Service name reported with the traces.
EI_METRICS_PORT	no	9464	Port for the Prometheus endpoint, 0 disables it.
RUST_LOG	no	eiapi=debug,tower_http=debug	Standard EnvFilter directive.

EI_DATA_DIR and its subdirectories are created at startup. Do not point it at /tmp: the retry and timeout queues have to survive a reboot.

Development

just check runs cargo fmt --check, cargo clippy -D warnings and the tests; see the justfile for the individual recipes.

Example

EI_PARSER_BIN=/opt/GW2-Elite-Insights-Parser/GW2EIParserCLI/out/GuildWars2EliteInsights-CLI \
EI_PARSER_SETTINGS=/opt/GW2-Elite-Insights-Parser/GW2EIParser/settings.conf \
EI_DATA_DIR=/var/lib/eiapi \
cargo run --release

Concurrency

EI_PARSER_JOBS parser processes run at the same time inside the one process; a dispatcher only takes a job off the queue when a parser slot is free, so queue_size and the position in GET /status/:ticket count everything that is still waiting. A file at position p waits for roughly p / workers parser runs.

Run one instance. Ticket numbers, the queue and the duplicate check all live in process memory, so putting several instances behind a load balancer means a client polling /status/:ticket usually asks an instance that never saw that ticket. Scale with EI_PARSER_JOBS, not with more processes. Each parser run is a full Elite Insights invocation, so this multiplies memory use too.

The parser settings file must have SaveOutTrace=true. The wingman response body only exists in the per-log trace file; without it a "wingman is full" refusal is indistinguishable from "this log is unsupported" and the log gets dropped. A missing trace is logged as a warning on every refusal.

Uploads

The file field must carry a filename; requests without one are rejected with 400. The stored name is <account without dots>_<client filename> and nothing is ever generated or mangled, so the client can predict it and match the log against wingman later. Filenames containing separators or control characters, or longer than 128 bytes, are rejected.

The body is streamed to $EI_DATA_DIR/tmp as it arrives and moved into uploads/ only once the whole request has been read, so a 200MB log costs a buffer, not 200MB of RAM. Aborted or rejected uploads delete their temp file.

A request whose name is already taken by a queued or parked log gets 409 instead of overwriting it, as does a repeat of the same (filesize, triggerID, account) within 24h. The duplicate filter is in memory and expires, so it does not grow without bound and does not survive a restart.

Shutdown and restarts

On SIGTERM or SIGINT the listener stops accepting requests, the dispatcher stops taking logs off the queue, and the process waits for the parser runs that are already in flight before exiting. Anything still queued stays in uploads/ and is requeued by the next start, so a restart never loses a log.

Startup does not wait for the parser update check when a parser is already installed, so the listener is back within about a tenth of a second; the check runs in the background. The remaining gap during a restart is the drain on the old container plus the container create. An interrupted parse is not lost. The file stays in uploads/ and is requeued, so there is no reason to raise stop_grace_period much.

Crash recovery

Only one instance may run, so at startup nothing in the data dir can belong to a live upload. On boot the server therefore:

• deletes everything in tmp/ (half-received uploads),
• deletes stale parser output (.log, .json, .html, .zip) and zero-length reserved names left in uploads/,
• requeues every remaining log in uploads/ under a fresh ticket.

Files in retry/ are left alone; they are drained by the next successful upload.

File lifecycle

• Uploads land in $EI_DATA_DIR/uploads (via $EI_DATA_DIR/tmp).
• Successfully processed files and their generated files are deleted.
• Logs wingman cannot take right now go to $EI_DATA_DIR/retry (see below). Everything in there is requeued automatically, so the directory being non-empty always means "waiting for wingman" and nothing else.
• $EI_DATA_DIR/dead holds files worth looking at: the parser crashed, produced no result, produced one that could not be read, or wingman refused the log EI_RETRY_MAX_ATTEMPTS times. Nothing leaves that directory on its own.
• Parser timeouts go to $EI_DATA_DIR/timeouts; a second run may well work.
• A log the parser read and rejected as invalid (too short, no targets, not a zip) is deleted. It will not become valid, and there is no parser bug to reproduce. The counter and the log line are the record. Same for a log wingman takes and permanently rejects.

Wingman capacity handling

Elite Insights reports every refusal as a single wingmanUploadRefused: true, whether the log is unsupported, already uploaded, or wingman is simply out of capacity. The reason only appears in the trace file the parser lists in generatedFiles:

Wingman: CheckUploadPossible successful: ERR_SERVER_CAPACITY_CRITICAL

The server reads that line and splits refusals in two:

• Transient (ERR_*, connection failures, wingman down): the evtc is parked in retry/ next to a <file>.retry.json sidecar and the ticket goes to deferred. Nothing is lost and the client does not need to re-upload.
• Permanent (False, anonymised logs): the file is deleted and the ticket goes to skipped, as before.

The backlog is not drained on a timer. The next upload that wingman accepts is the proof that capacity is back, so a successful upload immediately requeues every parked log, reusing the original ticket numbers. GET /status reports the current backlog size.