Skip to content

Possible regression: raw ANSI SGR fragments in assistant chat responses #1138

Open

Description

@magicprinc

Summary

Assistant chat responses sometimes display raw ANSI SGR fragments instead of formatting. This appears similar to closed issue #1106, but has been observed with Command Code 1.88.5 installed.

Example

A numbered list item containing bold text and inline code appeared like this (sanitized):

1. [1mExample bold text[22m ([38;2;177;186;249mExample.java:1-2[39m)

Expected: bold text and inline code render with formatting, without visible SGR fragments.

Environment

  • Command Code: 1.88.5
  • Installed marked-terminal: 7.3.1
  • OS: Windows
  • Terminal and shell: not confirmed

Notes

The report concerns assistant chat responses, not shell command output. Exact reproduction steps and whether this is the same underlying cause as #1106 are not yet confirmed.

Activity

  1. magicprinc commented on Nov 13, 2026

    @magicprinc
    Author

    BTW: I have benchmarked several harnesses with the same official API: OpenAI GPT-6.1 Terra HIGH (plan) and DeepSeek V4.1 Flash HIGH (build, test, review).

    Command Code, DSH (DeepSeek Harness), OpenCode v2, omp (oh my pi)

    Hard Java 25 task: MVCC Transactional Store

    💵 The cheapest development cost: DSH

    🚀 The best quality solution according to GPT‑6.1 Sol HIGH: Command Code

    I hope this helps:

    Verdict: Strong architecture and implementation, with one reproducible exceptional-path atomicity defect. Good unit tests, but not yet sufficient for a production-critical transactional component.

    Scoring scale: 1 = best, 10 = worst.

    Category Score Assessment
    Production code 2/10 Very good: compact, understandable, correct under normal MVCC scenarios; one important exceptional-path defect
    Unit tests 3/10 Good: broad semantic coverage; missing fault injection and stronger stress/property testing
    Architecture / planning 2/10 Conservative, well-structured snapshot-isolation design, faithfully implemented

    1. Assignment and solution

    The task was to design and implement an in-memory MvccStore<K, V> supporting snapshot isolation, read-your-writes, first-committer-wins write conflicts, atomic multi-key commits, tombstone-based deletion, transaction lifecycle, and safe history compaction. Full serializability was not required.

    The submission contains 280 lines of production Java in two files:

    • src/main/java/dev/mvcc/MvccStore.java — 251 lines.
    • src/main/java/dev/mvcc/WriteConflictException.java — 29 lines.

    The architecture uses a single monitor for synchronization, per-key version histories (Map<K, List<Version<V>>>), a TreeMap of active snapshot reference counts, and per-transaction write sets. Committed versions are monotonically increasing. Compaction preserves the history required by the oldest active snapshot.

    Strengths:

    • Easy-to-audit synchronization strategy; no unnecessary lock-free machinery.
    • Correct snapshot reads, read-your-writes, tombstones, and first-committer-wins behavior in tested normal cases.
    • Read-only transactions do not conflict with writers or advance the version.
    • Write skew is allowed, appropriately, under snapshot isolation.
    • Well-defined transaction states and cleanup through AutoCloseable.
    • Minimal dependencies and relatively small implementation.

    Tradeoffs: A single global monitor limits concurrent throughput. Reading an old snapshot scans a key's retained version history linearly. Compaction is manual, and abandoned transactions can retain history indefinitely. Keys and values are caller-owned and must obey the documented stability contract.

    2. Verification methodology and results

    The review examined PLAN.md, production source, all three submitted test classes, and BENCHMARK_REPORT.md. It also used independent checks beyond the agent's own test suite.

    Check Result Provenance
    Official Gradle/JUnit run 29 passed, 0 failed Reported by the submission, Java 25 / Gradle 9.9.1
    Independent compilation Passed OpenJDK 21, not Java 25
    Submitted tests through lightweight JUnit-compatible runner 29 passed, 0 failed Independent check; not the official JUnit engine
    Seeded reference-model exercise Passed 35,000 operations, including 10,572 reads, 438 conflicts, and 1,618 compactions
    Concurrent commit/compaction exercise Passed 12 worker threads, 8,400 attempted transactions
    Exceptional multi-key commit atomicity Failed Reproduced with a key whose hashCode() throws during publication

    The reference-model and concurrent checks found no normal-path snapshot or write-conflict inconsistencies. Passing these checks is not a proof of linearizability or freedom from all races. Independent verification did not reproduce the submission's exact Java 25/Gradle environment.

    3. Confirmed defect: exceptional multi-key commit is not atomic

    Severity: Moderate, exceptional path.
    Location: MvccStore.java, approximately lines 202–212.

    The commit operation increments the version and then publishes each key individually:

    long commitVersion = ++version;
    for (Map.Entry<K, PendingWrite<V>> entry : writes.entrySet()) {
        PendingWrite<V> write = entry.getValue();
        histories.computeIfAbsent(entry.getKey(), ignored -> new ArrayList<>())
                .add(new Version<>(commitVersion, write.deleted(), write.value()));
    }
    finish(State.COMMITTED);

    If a user-defined key's hashCode() throws partway through that loop, earlier keys have already been published. In the independent fault-injection case, the observed state was:

    currentVersion = 1
    first key      = Optional[1]
    second key     = Optional.empty
    transaction    = ACTIVE
    

    The store therefore violates an unconditional all-or-nothing multi-key commit guarantee under this exception. The issue was not seen with ordinary immutable, nonthrowing keys.

    Recommended resolution: Define the key contract explicitly and make the publication phase exception-safe, such as by preparing changes before an atomic state publication. Do not regard a preliminary hash check alone as a guarantee that later user callbacks cannot throw. Add a deterministic regression test using fault-injecting key behavior.

    4. Unit-test assessment — 3/10

    The submission has 29 JUnit 5 tests:

    Test class Tests Key areas
    MvccStoreCoreTest 17 Lifecycle, snapshot semantics, read-your-writes, conflict handling, nulls, write skew, overflow
    MvccStoreCompactionTest 7 Retention for active snapshots, tombstones, multiple snapshots, conflict history
    MvccStoreConcurrencyTest 5 Concurrent writers, read-only transactions, atomic visibility, rollback, compaction races
    Total 29 All passed in the submitted final run

    What the tests do well: They cover transaction semantics rather than only happy paths. Particularly valuable cases include write skew, deleting/recreating keys, compaction with multiple active snapshots, concurrent competing commits, and read-only transactions alongside writers.

    Missing or insufficient coverage:

    1. Exception/fault injection: a failing key hashCode() or equals() during commit; the confirmed defect escaped the suite.
    2. Reference-model/property testing: randomly generated transaction histories against an independently implemented oracle.
    3. More systematic concurrency exploration: repeated high-contention runs and deliberately coordinated rare interleavings.
    4. Performance/regression testing: many retained versions, long-lived snapshots, and compaction overhead.

    The tests are strong for an AI-generated benchmark submission, but 29 passing tests cannot establish that the implementation is bug-free.

    5. Plan → Build quality and efficiency

    Terra selected a compact global-lock MVCC design. DeepSeek implemented that design closely, with no unnecessary frameworks. PLAN.md remained unchanged after the handoff, according to the submitted report.

    Metric Submitted value
    PLAN duration 3m 25s
    BUILD duration 2m 23s
    VERIFY & FIX duration 3m 49s
    PLAN + BUILD + VERIFY measured windows 9m 37s
    Terra calls 9
    Terra input / output tokens 557,607 / 1,754
    Terra cached-read tokens 72,320
    Terra reported API cost $0.497673
    DeepSeek calls 42
    DeepSeek API cost Not recorded

    These phase windows exclude some model handoff and report-writing overhead. The transcript's first Terra call to its last reported DeepSeek record spans approximately 17m 05s. The transcript's inputTokens may include cached input; these values must not be added together as separate token categories.

    The plan was approximately 380 words, exceeding the 350-word planning limit. For a budget-constrained workflow, Terra's approximately $0.50 reported cost for planning and associated handoff turns is a meaningful concern. The automatic handoff itself worked as intended in the submitted transcript; the exact underlying DeepSeek release is confirmed by the operator, rather than resolved from the provider alias alone.

    6. Final judgment (1 = the best)

    Production implementation: 2/10. Unit tests: 3/10. On the stated scale, these are strong ratings, not failing grades.

    Command Code demonstrated effective Terra HIGH planning → DeepSeek V4.1 Flash HIGH implementation and verification. The result is compact, maintainable, and passed extensive normal-case MVCC checks. The main remaining correctness problem is exceptional commit atomicity; the principal engineering tradeoff is coarse-grained locking.

    Disposition: Good benchmark result and credible foundation for further development. Not approved as bug-free, production-critical transactional code until the atomicity defect is addressed and additional adversarial tests pass.

    Limits of this evaluation

    This was one task and one run, not a statistically significant comparison of coding harnesses or AI models. The reported model costs and timing come from the submission, while the additional checks were conducted independently. The independent JUnit-compatible runner and Java 21 compilation do not substitute for an independently reproduced Java 25 build.

  2. naymurdev commented on Nov 13, 2026

    @naymurdev

    @vipulgupta2048 can u have a look at this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions