Chu-former

<p><span>Semantic handshake between questions and answers </span></p> <p> </p> <p><span>Data: n queries qi, m key-value pairs (kj, vj)</span></p> <p> </p> <p><span>Let score be a similarity function between keys and queries.</span></p> <p><span>Fix j, let aij = soft max score(kj, qi) be probabilities over the n queries.</span></p> <p><span>Fix i, let bij = soft max score (kj, qi) be probabilities over the m keys and j* = arg max bij.</span></p> <p><span>Let H(i, α) be the Renyi α-entropy of {bij} and </span></p> <p><span>E({qi}) = (1/n) Sum over i [ H(i, α) ] be the average entropy.</span></p> <p> </p> <p><span>Learn NN p: K -> V such that p(kj) = vj and </span></p> <p><span>learn NN d: K -> Q such that d(kj) = Sum over i [ aij x qi ].</span></p> <p> </p> <p><span>For semantics d: K -> Q and p: K -> V, if the commutative square diagram given by,</span></p> <p><span>(1 x d) o (p x 1) = (p x1) o (1 x d) has a diagonal lifting τ: K x Q -> V x K, </span></p> <p><span>learn NN c: Q -> K such that c(qi) = proj2(τ (kj*, qi)), then d(c(qi)) = qi, </span></p> <p><span>that is c is a section of d, a form of abductive reasoning.</span></p> <p><span>In summary, d and p are Chu adjoints under τ, and τ generates a section c of d.</span></p> <p> </p> <p><span>Similarly, if (1 x p) o (d x 1) = (d x 1) o (1 x p) has a diagonal lifting σ: K x V -> Q x K,</span></p> <p><span>learn NN s: V -> K such that s(vj) = proj2(σ (kj, vj)), then p(s(vj)) = vj,</span></p> <p><span>that is s is a section of p, a form of abduction.<span>   </span></span></p> <p><span>So we have a semantic bridge c: Q -> K <- V :s</span></p> <p> </p> <p><span>Let m: K x K -> PositiveReals be a metric on keys.</span></p> <p><span>Let s(j^) be the key closest to c(qi) under metric m; a key-handshake between query and value.<span>  </span>In reply to query qi, answer value vj^.</span></p> <p> </p> <p><span>Query Updating </span></p> <p><span>Let query update qi’ = Sum over j [ bij x vj ] and d’ be the corresponding query semantics, </span></p> <p><span>and let τ’(d’) and σ’(d’) be the corresponding liftings.</span></p> <p><span>If E({qi’}) < E({qi}) assign qi := qi’, τ := τ’, σ := σ’ and repeat.</span></p> <p><span>Stop updating when entropy is not being reduced or is sufficiently small, at which point output value vj^ in answer to query qi.</span></p> <p> </p> <p><span>Summary </span></p> <p><span>Keys provide semantics for values and queries via learned neural networks p: K -> V and d: K -> Q.<span>  </span>p and d are Chu duals with respect to fibration liftings τ and σ that provide sections c: Q -> K <- V :s creating a semantic bridge from queries to values.</span></p> <p><span>Query/lifting updating is achieved by entropy reduction.</span></p>