2
Module difflib -- helpers for computing deltas between objects.
4
Function get_close_matches(word, possibilities, n=3, cutoff=0.6):
5
Use SequenceMatcher to return list of the best "good enough" matches.
7
Function context_diff(a, b):
8
For two lists of strings, return a delta in context diff format.
11
Return a delta: the difference between `a` and `b` (lists of strings).
13
Function restore(delta, which):
14
Return one of the two sequences that generated an ndiff delta.
16
Function unified_diff(a, b):
17
For two lists of strings, return a delta in unified diff format.
19
Class SequenceMatcher:
20
A flexible class for comparing pairs of sequences of any type.
23
For producing human-readable deltas from sequences of lines of text.
26
For producing HTML side by side comparison with change highlights.
29
__all__ = ['get_close_matches', 'ndiff', 'restore', 'SequenceMatcher',
30
'Differ','IS_CHARACTER_JUNK', 'IS_LINE_JUNK', 'context_diff',
31
'unified_diff', 'diff_bytes', 'HtmlDiff', 'Match']
33
from heapq import nlargest as _nlargest
34
from collections import namedtuple as _namedtuple
35
from types import GenericAlias
36
lazy from _colorize import can_colorize, get_theme
38
Match = _namedtuple('Match', 'a b size')
40
def _calculate_ratio(matches, length):
42
return 2.0 * matches / length
45
class SequenceMatcher:
48
SequenceMatcher is a flexible class for comparing pairs of sequences of
49
any type, so long as the sequence elements are hashable. The basic
50
algorithm predates, and is a little fancier than, an algorithm
51
published in the late 1980's by Ratcliff and Obershelp under the
52
hyperbolic name "gestalt pattern matching". The basic idea is to find
53
the longest contiguous matching subsequence that contains no "junk"
54
elements (R-O doesn't address junk). The same idea is then applied
55
recursively to the pieces of the sequences to the left and to the right
56
of the matching subsequence. This does not yield minimal edit
57
sequences, but does tend to yield matches that "look right" to people.
59
SequenceMatcher tries to compute a "human-friendly diff" between two
60
sequences. Unlike e.g. UNIX(tm) diff, the fundamental notion is the
61
longest *contiguous* & junk-free matching subsequence. That's what
62
catches peoples' eyes. The Windows(tm) windiff has another interesting
63
notion, pairing up elements that appear uniquely in each sequence.
64
That, and the method here, appear to yield more intuitive difference
65
reports than does diff. This method appears to be the least vulnerable
66
to syncing up on blocks of "junk lines", though (like blank lines in
67
ordinary text files, or maybe "<P>" lines in HTML files). That may be
68
because this is the only method of the 3 that has a *concept* of
71
Example, comparing two strings, and considering blanks to be "junk":
73
>>> s = SequenceMatcher(lambda x: x == " ",
74
... "private Thread currentThread;",
75
... "private volatile Thread currentThread;")
78
.ratio() returns a float in [0, 1], measuring the "similarity" of the
79
sequences. As a rule of thumb, a .ratio() value over 0.6 means the
80
sequences are close matches:
82
>>> print(round(s.ratio(), 2))
86
If you're only interested in where the sequences match,
87
.get_matching_blocks() is handy:
89
>>> for block in s.get_matching_blocks():
90
... print("a[%d] and b[%d] match for %d elements" % block)
91
a[0] and b[0] match for 8 elements
92
a[8] and b[17] match for 21 elements
93
a[29] and b[38] match for 0 elements
95
Note that the last tuple returned by .get_matching_blocks() is always a
96
dummy, (len(a), len(b), 0), and this is the only case in which the last
97
tuple element (number of elements matched) is 0.
99
If you want to know how to change the first sequence into the second,
102
>>> for opcode in s.get_opcodes():
103
... print("%6s a[%d:%d] b[%d:%d]" % opcode)
105
insert a[8:8] b[8:17]
106
equal a[8:29] b[17:38]
108
See the Differ class for a fancy human-friendly file differencer, which
109
uses SequenceMatcher both to compare sequences of lines, and to compare
110
sequences of characters within similar (near-matching) lines.
112
See also function get_close_matches() in this module, which shows how
113
simple code building on SequenceMatcher can be used to do useful work.
115
Timing: Basic R-O is cubic time worst case and quadratic time expected
116
case. SequenceMatcher is quadratic time for the worst case and has
117
expected-case behavior dependent in a complicated way on how many
118
elements the sequences have in common; best case time is linear.
121
def __init__(self, isjunk=None, a='', b='', autojunk=True):
122
"""Construct a SequenceMatcher.
124
Optional arg isjunk is None (the default), or a one-argument
125
function that takes a sequence element and returns true iff the
126
element is junk. None is equivalent to passing "lambda x: 0", i.e.
127
no elements are considered to be junk. For example, pass
128
lambda x: x in " \\t"
129
if you're comparing lines as sequences of characters, and don't
130
want to synch up on blanks or hard tabs.
132
Optional arg a is the first of two sequences to be compared. By
133
default, an empty string. The elements of a must be hashable. See
134
also .set_seqs() and .set_seq1().
136
Optional arg b is the second of two sequences to be compared. By
137
default, an empty string. The elements of b must be hashable. See
138
also .set_seqs() and .set_seq2().
140
Optional arg autojunk should be set to False to disable the
141
"automatic junk heuristic" that treats popular elements as junk
142
(see module documentation for more information).
149
# second sequence; differences are computed as "what do
150
# we need to do to 'a' to change it into 'b'?"
152
# for x in b, b2j[x] is a list of the indices (into b)
153
# at which x appears; junk and popular elements do not appear
155
# for x in b, fullbcount[x] == the number of times x
156
# appears in b; only materialized if really needed (used
157
# only for computing quick_ratio())
159
# a list of (i, j, k) triples, where a[i:i+k] == b[j:j+k];
160
# ascending & non-overlapping in i and in j; terminated by
161
# a dummy (len(a), len(b), 0) sentinel
163
# a list of (tag, i1, i2, j1, j2) tuples, where tag is
165
# 'replace' a[i1:i2] should be replaced by b[j1:j2]
166
# 'delete' a[i1:i2] should be deleted
167
# 'insert' b[j1:j2] should be inserted
168
# 'equal' a[i1:i2] == b[j1:j2]
170
# a user-supplied function taking a sequence element and
171
# returning true iff the element is "junk" -- this has
172
# subtle but helpful effects on the algorithm, which I'll
173
# get around to writing up someday <0.9 wink>.
174
# DON'T USE! Only __chain_b uses this. Use "in self.bjunk".
176
# the items in b for which isjunk is True.
178
# nonjunk items in b treated as junk by the heuristic (if used).
181
self.a = self.b = None
182
self.autojunk = autojunk
183
499
self.set_seqs(a, b)
185
def set_seqs(self, a, b):
186
"""Set the two sequences to be compared.
188
>>> s = SequenceMatcher()
189
>>> s.set_seqs("abcd", "bcde")
195
499
self.set_seq2(b)
197
def set_seq1(self, a):
198
"""Set the first sequence to be compared.
200
The second sequence to be compared is not changed.
202
>>> s = SequenceMatcher(None, "abcd", "bcde")
205
>>> s.set_seq1("bcde")
210
SequenceMatcher computes and caches detailed information about the
211
second sequence, so if you want to compare one sequence S against
212
many sequences, use .set_seq2(S) once and call .set_seq1(x)
213
repeatedly for each of the other sequences.
215
See also set_seqs() and set_seq2().
221
self.matching_blocks = self.opcodes = None
223
def set_seq2(self, b):
224
"""Set the second sequence to be compared.
226
The first sequence to be compared is not changed.
228
>>> s = SequenceMatcher(None, "abcd", "bcde")
231
>>> s.set_seq2("abcd")
236
SequenceMatcher computes and caches detailed information about the
237
second sequence, so if you want to compare one sequence S against
238
many sequences, use .set_seq2(S) once and call .set_seq1(x)
239
repeatedly for each of the other sequences.
241
See also set_seqs() and set_seq1().
247
self.matching_blocks = self.opcodes = None
248
self.fullbcount = None
249
499
self.__chain_b()
251
# For each element x in b, set b2j[x] to a list of the indices in
252
# b where x appears; the indices are in increasing order; note that
253
# the number of times x appears in b is len(b2j[x]) ...
254
# when self.isjunk is defined, junk elements don't show up in this
255
# map at all, which stops the central find_longest_match method
256
# from starting any matching block at a junk element ...
257
# b2j also does not contain entries for "popular" elements, meaning
258
# elements that account for more than 1 + 1% of the total elements, and
259
# when the sequence is reasonably large (>= 200 elements); this can
260
# be viewed as an adaptive notion of semi-junk, and yields an enormous
261
# speedup when, e.g., comparing program files with hundreds of
262
# instances of "return NULL;" ...
263
# note that this is only called when b changes; so for cross-product
264
# kinds of matches, it's best to call set_seq2 once, then set_seq1
268
# Because isjunk is a user-defined (not C) function, and we test
269
# for junk a LOT, it's important to minimize the number of calls.
270
# Before the tricks described here, __chain_b was by far the most
271
# time-consuming routine in the whole module! If anyone sees
272
# Jim Roskind, thank him again for profile.py -- I never would
274
# The first trick is to build b2j ignoring the possibility
275
# of junk. I.e., we don't call isjunk at all yet. Throwing
276
# out the junk later is much cheaper than building b2j "right"
281
36
37
for i, elt in enumerate(b):
282
220
372
indices = b2j.setdefault(elt, [])
283
27
28
indices.append(i)
285
# Purge junk elements
286
self.bjunk = junk = set()
289
for elt in b2j.keys():
292
for elt in junk: # separate loop avoids separate list of keys
295
# Purge popular elements that are not junk
296
self.bpopular = popular = set()
298
if self.autojunk and n >= 200:
300
29
30
for elt, idxs in b2j.items():
301
32
32
if len(idxs) > ntest:
303
for elt in popular: # ditto; as fast for 1% deletion
306
def find_longest_match(self, alo=0, ahi=None, blo=0, bhi=None):
307
"""Find longest matching block in a[alo:ahi] and b[blo:bhi].
309
By default it will find the longest match in the entirety of a and b.
311
If isjunk is not defined:
313
Return (i,j,k) such that a[i:i+k] is equal to b[j:j+k], where
314
alo <= i <= i+k <= ahi
315
blo <= j <= j+k <= bhi
316
and for all (i',j',k') meeting those conditions,
319
and if i == i', j <= j'
321
In other words, of all maximal matching blocks, return one that
322
starts earliest in a, and of all those maximal matching blocks that
323
start earliest in a, return the one that starts earliest in b.
325
>>> s = SequenceMatcher(None, " abcd", "abcd abcd")
326
>>> s.find_longest_match(0, 5, 0, 9)
327
Match(a=0, b=4, size=5)
329
If isjunk is defined, first the longest matching block is
330
determined as above, but with the additional restriction that no
331
junk element appears in the block. Then that block is extended as
332
far as possible by matching (only) junk elements on both sides. So
333
the resulting block never matches on junk except as identical junk
334
happens to be adjacent to an "interesting" match.
336
Here's the same example as before, but considering blanks to be
337
junk. That prevents " abcd" from matching the " abcd" at the tail
338
end of the second sequence directly. Instead only the "abcd" can
339
match, and matches the leftmost "abcd" in the second sequence:
341
>>> s = SequenceMatcher(lambda x: x==" ", " abcd", "abcd abcd")
342
>>> s.find_longest_match(0, 5, 0, 9)
343
Match(a=1, b=0, size=4)
345
If no blocks match, return (alo, blo, 0).
347
>>> s = SequenceMatcher(None, "ab", "c")
348
>>> s.find_longest_match(0, 2, 0, 1)
349
Match(a=0, b=0, size=0)
352
# CAUTION: stripping common prefix or suffix would be incorrect.
356
# Longest matching block is "ab", but if common prefix is
357
# stripped, it's "a" (tied with "b"). UNIX(tm) diff does so
358
# strip, so ends up claiming that ab is changed to acab by
359
# inserting "ca" in the middle. That's minimal but unintuitive:
360
# "it's obvious" that someone inserted "ac" at the front.
361
# Windiff ends up at the same place as diff, but by pairing up
362
# the unique 'b's and then matching the first two 'a's.
364
2
2
a, b, b2j, isbjunk = self.a, self.b, self.b2j, self.bjunk.__contains__
369
besti, bestj, bestsize = alo, blo, 0
370
# find longest junk-free match
371
# during an iteration of the loop, j2len[j] = length of longest
372
# junk-free match ending with a[i-1] and b[j]
375
197
197
for i in range(alo, ahi):
376
# look at all instances of a[i] in b; note that because
377
# b2j has no junk keys, the loop is skipped if a[i] is junk
378
1104
1104
j2lenget = j2len.get
379
225
225
newj2len = {}
380
1916
1916
for j in b2j.get(a[i], nothing):
382
399
399
if j < blo:
384
113
113
if j >= bhi:
386
1386
1386
k = newj2len[j] = j2lenget(j-1, 0) + 1
387
334
334
if k > bestsize:
388
75
75
besti, bestj, bestsize = i-k+1, j-k+1, k
389
96
96
j2len = newj2len
391
# Extend the best by non-junk elements on each end. In particular,
392
# "popular" non-junk elements aren't in b2j, which greatly speeds
393
# the inner loop above, but also means "the best" match so far
394
# doesn't contain any junk *or* popular non-junk elements.
395
2
2
while besti > alo and bestj > blo and \
396
not isbjunk(b[bestj-1]) and \
397
a[besti-1] == b[bestj-1]:
398
besti, bestj, bestsize = besti-1, bestj-1, bestsize+1
399
while besti+bestsize < ahi and bestj+bestsize < bhi and \
400
1
1
not isbjunk(b[bestj+bestsize]) and \
401
a[besti+bestsize] == b[bestj+bestsize]:
404
# Now that we have a wholly interesting match (albeit possibly
405
# empty!), we may as well suck up the matching junk on each
406
# side of it too. Can't think of a good reason not to, and it
407
# saves post-processing the (possibly considerable) expense of
408
# figuring out what to do with it. In the case of an empty
409
# interesting match, this is clearly the right thing to do,
410
# because no other kind of match is possible in the regions.
411
while besti > alo and bestj > blo and \
412
isbjunk(b[bestj-1]) and \
413
a[besti-1] == b[bestj-1]:
414
besti, bestj, bestsize = besti-1, bestj-1, bestsize+1
415
while besti+bestsize < ahi and bestj+bestsize < bhi and \
416
1
1
isbjunk(b[bestj+bestsize]) and \
417
a[besti+bestsize] == b[bestj+bestsize]:
418
bestsize = bestsize + 1
420
3
3
return Match(besti, bestj, bestsize)
422
def get_matching_blocks(self):
423
"""Return list of triples describing matching subsequences.
425
Each triple is of the form (i, j, n), and means that
426
a[i:i+n] == b[j:j+n]. The triples are monotonically increasing in
427
i and in j. New in Python 2.5, it's also guaranteed that if
428
(i, j, n) and (i', j', n') are adjacent triples in the list, and
429
the second is not the last triple in the list, then i+n != i' or
430
j+n != j'. IOW, adjacent triples never describe adjacent equal
433
The last triple is a dummy, (len(a), len(b), 0), and is the only
436
>>> s = SequenceMatcher(None, "abxcd", "abcd")
437
>>> list(s.get_matching_blocks())
438
[Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)]
441
if self.matching_blocks is not None:
442
return self.matching_blocks
443
la, lb = len(self.a), len(self.b)
445
# This is most naturally expressed as a recursive algorithm, but
446
# at least one user bumped into extreme use cases that exceeded
447
# the recursion limit on their box. So, now we maintain a list
448
# ('queue`) of blocks we still need to look at, and append partial
449
# results to `matching_blocks` in a loop; the matches are sorted
451
queue = [(0, la, 0, lb)]
454
alo, ahi, blo, bhi = queue.pop()
455
5
5859
i, j, k = x = self.find_longest_match(alo, ahi, blo, bhi)
456
# a[alo:i] vs b[blo:j] unknown
457
# a[i:i+k] same as b[j:j+k]
458
# a[i+k:ahi] vs b[j+k:bhi] unknown
459
if k: # if k is 0, there was no matching block
460
matching_blocks.append(x)
461
if alo < i and blo < j:
462
queue.append((alo, i, blo, j))
463
if i+k < ahi and j+k < bhi:
464
queue.append((i+k, ahi, j+k, bhi))
465
matching_blocks.sort()
467
# It's possible that we have adjacent equal blocks in the
468
# matching_blocks list now. Starting with 2.5, this code was added
472
for i2, j2, k2 in matching_blocks:
473
# Is this block adjacent to i1, j1, k1?
474
if i1 + k1 == i2 and j1 + k1 == j2:
475
# Yes, so collapse them -- this just increases the length of
476
# the first block by the length of the second, and the first
477
# block so lengthened remains the block to compare against.
480
# Not adjacent. Remember the first block (k1==0 means it's
481
# the dummy we started with), and make the second block the
482
# new block to compare against.
484
non_adjacent.append((i1, j1, k1))
485
i1, j1, k1 = i2, j2, k2
487
non_adjacent.append((i1, j1, k1))
489
non_adjacent.append( (la, lb, 0) )
490
2
2
self.matching_blocks = list(map(Match._make, non_adjacent))
491
return self.matching_blocks
493
def get_opcodes(self):
494
"""Return list of 5-tuples describing how to turn a into b.
496
Each tuple is of the form (tag, i1, i2, j1, j2). The first tuple
497
has i1 == j1 == 0, and remaining tuples have i1 == the i2 from the
498
tuple preceding it, and likewise for j1 == the previous j2.
500
The tags are strings, with these meanings:
502
'replace': a[i1:i2] should be replaced by b[j1:j2]
503
'delete': a[i1:i2] should be deleted.
504
Note that j1==j2 in this case.
505
'insert': b[j1:j2] should be inserted at a[i1:i1].
506
Note that i1==i2 in this case.
507
'equal': a[i1:i2] == b[j1:j2]
511
>>> s = SequenceMatcher(None, a, b)
512
>>> for tag, i1, i2, j1, j2 in s.get_opcodes():
513
... print(("%7s a[%d:%d] (%s) b[%d:%d] (%s)" %
514
... (tag, i1, i2, a[i1:i2], j1, j2, b[j1:j2])))
515
delete a[0:1] (q) b[0:0] ()
516
equal a[1:3] (ab) b[0:2] (ab)
517
replace a[3:4] (x) b[2:3] (y)
518
equal a[4:6] (cd) b[3:5] (cd)
519
insert a[6:6] () b[5:6] (f)
522
if self.opcodes is not None:
525
self.opcodes = answer = []
526
for ai, bj, size in self.get_matching_blocks():
527
# invariant: we've pumped out correct diffs to change
528
# a[:i] into b[:j], and the next matching block is
529
# a[ai:ai+size] == b[bj:bj+size]. So we need to pump
530
# out a diff to change a[i:ai] into b[j:bj], pump out
531
# the matching block, and move (i,j) beyond the match
533
if i < ai and j < bj:
540
answer.append( (tag, i, ai, j, bj) )
541
i, j = ai+size, bj+size
542
# the list of matching blocks is terminated by a
543
# sentinel with size 0
545
answer.append( ('equal', ai, i, bj, j) )
548
def get_grouped_opcodes(self, n=3):
549
""" Isolate change clusters by eliminating ranges with no changes.
551
Return a generator of groups with up to n lines of context.
552
Each group is in the same format as returned by get_opcodes().
554
>>> from pprint import pprint
555
>>> a = list(map(str, range(1,40)))
557
>>> b[8:8] = ['i'] # Make an insertion
558
>>> b[20] += 'x' # Make a replacement
559
>>> b[23:28] = [] # Make a deletion
560
>>> b[30] += 'y' # Make another replacement
561
>>> pprint(list(SequenceMatcher(None,a,b).get_grouped_opcodes()))
562
[[('equal', 5, 8, 5, 8), ('insert', 8, 8, 8, 9), ('equal', 8, 11, 9, 12)],
563
[('equal', 16, 19, 17, 20),
564
('replace', 19, 20, 20, 21),
565
('equal', 20, 22, 21, 23),
566
('delete', 22, 27, 23, 23),
567
('equal', 27, 30, 23, 26)],
568
[('equal', 31, 34, 27, 30),
569
('replace', 34, 35, 30, 31),
570
('equal', 35, 38, 31, 34)]]
573
codes = self.get_opcodes()
575
codes = [("equal", 0, 1, 0, 1)]
576
# Fixup leading and trailing groups if they show no changes.
577
if codes[0][0] == 'equal':
578
tag, i1, i2, j1, j2 = codes[0]
579
codes[0] = tag, max(i1, i2-n), i2, max(j1, j2-n), j2
580
if codes[-1][0] == 'equal':
581
tag, i1, i2, j1, j2 = codes[-1]
582
codes[-1] = tag, i1, min(i2, i1+n), j1, min(j2, j1+n)
586
for tag, i1, i2, j1, j2 in codes:
587
# End the current group and start a new one whenever
588
# there is a large range with no changes.
589
if tag == 'equal' and i2-i1 > nn:
590
group.append((tag, i1, min(i2, i1+n), j1, min(j2, j1+n)))
593
i1, j1 = max(i1, i2-n), max(j1, j2-n)
594
group.append((tag, i1, i2, j1 ,j2))
595
if group and not (len(group)==1 and group[0][0] == 'equal'):
599
"""Return a measure of the sequences' similarity (float in [0,1]).
601
Where T is the total number of elements in both sequences, and
602
M is the number of matches, this is 2.0*M / T.
603
Note that this is 1 if the sequences are identical, and 0 if
604
they have nothing in common.
606
.ratio() is expensive to compute if you haven't already computed
607
.get_matching_blocks() or .get_opcodes(), in which case you may
608
want to try .quick_ratio() or .real_quick_ratio() first to get an
611
>>> s = SequenceMatcher(None, "abcd", "bcde")
616
>>> s.real_quick_ratio()
620
matches = sum(triple[-1] for triple in self.get_matching_blocks())
621
return _calculate_ratio(matches, len(self.a) + len(self.b))
623
def quick_ratio(self):
624
"""Return an upper bound on ratio() relatively quickly.
626
This isn't defined beyond that it is an upper bound on .ratio(), and
627
is faster to compute.
630
# viewing a and b as multisets, set matches to the cardinality
631
# of their intersection; this counts the number of matches
632
# without regard to order, so is clearly an upper bound
633
if self.fullbcount is None:
634
self.fullbcount = fullbcount = {}
636
fullbcount[elt] = fullbcount.get(elt, 0) + 1
637
fullbcount = self.fullbcount
638
# avail[x] is the number of times x appears in 'b' less the
639
# number of times we've seen it in 'a' so far ... kinda
646
numb = fullbcount.get(elt, 0)
647
avail[elt] = numb - 1
650
return _calculate_ratio(matches, len(self.a) + len(self.b))
652
def real_quick_ratio(self):
653
"""Return an upper bound on ratio() very quickly.
655
This isn't defined beyond that it is an upper bound on .ratio(), and
656
is faster to compute than either .ratio() or .quick_ratio().
659
la, lb = len(self.a), len(self.b)
660
# can't have more matches than the number of elements in the
662
return _calculate_ratio(min(la, lb), la + lb)
664
__class_getitem__ = classmethod(GenericAlias)
667
def get_close_matches(word, possibilities, n=3, cutoff=0.6):
668
"""Use SequenceMatcher to return list of the best "good enough" matches.
670
word is a sequence for which close matches are desired (typically a
673
possibilities is a list of sequences against which to match word
674
(typically a list of strings).
676
Optional arg n (default 3) is the maximum number of close matches to
677
return. n must be > 0.
679
Optional arg cutoff (default 0.6) is a float in [0, 1]. Possibilities
680
that don't score at least that similar to word are ignored.
682
The best (no more than n) matches among the possibilities are returned
683
in a list, sorted by similarity score, most similar first.
685
>>> get_close_matches("appel", ["ape", "apple", "peach", "puppy"])
687
>>> import keyword as _keyword
688
>>> get_close_matches("wheel", _keyword.kwlist)
690
>>> get_close_matches("Apple", _keyword.kwlist)
692
>>> get_close_matches("accept", _keyword.kwlist)
697
raise ValueError("n must be > 0: %r" % (n,))
698
if not 0.0 <= cutoff <= 1.0:
699
raise ValueError("cutoff must be in [0.0, 1.0]: %r" % (cutoff,))
701
s = SequenceMatcher()
703
for x in possibilities:
705
if s.real_quick_ratio() < cutoff or s.quick_ratio() < cutoff:
710
result.append((ratio, x))
712
# Move the best scorers to head of list
713
result = _nlargest(n, result)
714
# Strip scores for the best n matches
715
return [x for score, x in result]
718
def _keep_original_ws(s, tag_s):
719
"""Replace whitespace with the original whitespace characters in `s`"""
721
c if tag_c == " " and c.isspace() else tag_c
722
for c, tag_c in zip(s, tag_s)
729
Differ is a class for comparing sequences of lines of text, and
730
producing human-readable differences or deltas. Differ uses
731
SequenceMatcher both to compare sequences of lines, and to compare
732
sequences of characters within similar (near-matching) lines.
734
Each line of a Differ delta begins with a two-letter code:
736
'- ' line unique to sequence 1
737
'+ ' line unique to sequence 2
738
' ' line common to both sequences
739
'? ' line not present in either input sequence
741
Lines beginning with '? ' attempt to guide the eye to intraline
742
differences, and were not present in either input sequence. These lines
743
can be confusing if the sequences contain tab characters.
745
Note that Differ makes no claim to produce a *minimal* diff. To the
746
contrary, minimal diffs are often counter-intuitive, because they synch
747
up anywhere possible, sometimes accidental matches 100 pages apart.
748
Restricting synch points to contiguous matches preserves some notion of
749
locality, at the occasional cost of producing a longer diff.
751
Example: Comparing two texts.
753
First we set up the texts, sequences of individual single-line strings
754
ending with newlines (such sequences can also be obtained from the
755
`readlines()` method of file-like objects):
757
>>> text1 = ''' 1. Beautiful is better than ugly.
758
... 2. Explicit is better than implicit.
759
... 3. Simple is better than complex.
760
... 4. Complex is better than complicated.
761
... '''.splitlines(keepends=True)
766
>>> text2 = ''' 1. Beautiful is better than ugly.
767
... 3. Simple is better than complex.
768
... 4. Complicated is better than complex.
769
... 5. Flat is better than nested.
770
... '''.splitlines(keepends=True)
772
Next we instantiate a Differ object:
776
Note that when instantiating a Differ object we may pass functions to
777
filter out line and character 'junk'. See Differ.__init__ for details.
779
Finally, we compare the two:
781
>>> result = list(d.compare(text1, text2))
783
'result' is a list of strings, so let's pretty-print it:
785
>>> from pprint import pprint as _pprint
787
[' 1. Beautiful is better than ugly.\n',
788
'- 2. Explicit is better than implicit.\n',
789
'- 3. Simple is better than complex.\n',
790
'+ 3. Simple is better than complex.\n',
792
'- 4. Complex is better than complicated.\n',
794
'+ 4. Complicated is better than complex.\n',
796
'+ 5. Flat is better than nested.\n']
798
As a single multi-line string it looks like this:
800
>>> print(''.join(result), end="")
801
1. Beautiful is better than ugly.
802
- 2. Explicit is better than implicit.
803
- 3. Simple is better than complex.
804
+ 3. Simple is better than complex.
806
- 4. Complex is better than complicated.
808
+ 4. Complicated is better than complex.
810
+ 5. Flat is better than nested.
813
def __init__(self, linejunk=None, charjunk=None):
815
Construct a text differencer, with optional filters.
817
The two optional keyword parameters are for filter functions:
819
- `linejunk`: A function that should accept a single string argument,
820
and return true iff the string is junk. The module-level function
821
`IS_LINE_JUNK` may be used to filter out lines without visible
822
characters, except for at most one splat ('#'). It is recommended
823
to leave linejunk None; the underlying SequenceMatcher class has
824
an adaptive notion of "noise" lines that's better than any static
825
definition the author has ever been able to craft.
827
- `charjunk`: A function that should accept a string of length 1. The
828
module-level function `IS_CHARACTER_JUNK` may be used to filter out
829
whitespace characters (a blank or tab; **note**: bad idea to include
830
newline in this!). Use of IS_CHARACTER_JUNK is recommended.
833
self.linejunk = linejunk
834
self.charjunk = charjunk
836
def compare(self, a, b):
838
Compare two sequences of lines; generate the resulting delta.
840
Each sequence must contain individual single-line strings ending with
841
newlines. Such sequences can be obtained from the `readlines()` method
842
of file-like objects. The delta generated also consists of newline-
843
terminated strings, ready to be printed as-is via the writelines()
844
method of a file-like object.
848
>>> print(''.join(Differ().compare('one\ntwo\nthree\n'.splitlines(True),
849
... 'ore\ntree\nemu\n'.splitlines(True))),
862
cruncher = SequenceMatcher(self.linejunk, a, b)
863
for tag, alo, ahi, blo, bhi in cruncher.get_opcodes():
865
g = self._fancy_replace(a, alo, ahi, b, blo, bhi)
866
elif tag == 'delete':
867
g = self._dump('-', a, alo, ahi)
868
elif tag == 'insert':
869
g = self._dump('+', b, blo, bhi)
871
g = self._dump(' ', a, alo, ahi)
873
raise ValueError('unknown tag %r' % (tag,))
877
def _dump(self, tag, x, lo, hi):
878
"""Generate comparison results for a same-tagged range."""
879
for i in range(lo, hi):
880
yield '%s %s' % (tag, x[i])
882
def _plain_replace(self, a, alo, ahi, b, blo, bhi):
883
assert alo < ahi and blo < bhi
884
# dump the shorter block first -- reduces the burden on short-term
885
# memory if the blocks are of very different sizes
886
if bhi - blo < ahi - alo:
887
first = self._dump('+', b, blo, bhi)
888
second = self._dump('-', a, alo, ahi)
890
first = self._dump('-', a, alo, ahi)
891
second = self._dump('+', b, blo, bhi)
893
for g in first, second:
896
def _fancy_replace(self, a, alo, ahi, b, blo, bhi):
898
When replacing one block of lines with another, search the blocks
899
for *similar* lines; the best-matching pair (if any) is used as a
900
synch point, and intraline difference marking is done on the
901
similar pair. Lots of work, but often worth it.
906
>>> results = d._fancy_replace(['abcDefghiJkl\n'], 0, 1,
907
... ['abcdefGhijkl\n'], 0, 1)
908
>>> print(''.join(results), end="")
914
# Don't synch up unless the lines have a similarity score above
915
# cutoff. Previously only the smallest pair was handled here,
916
# and if there are many pairs with the best ratio, recursion
917
# could grow very deep, and runtime cubic. See:
918
# https://github.com/python/cpython/issues/119105
920
# Later, more pathological cases prompted removing recursion
923
cruncher = SequenceMatcher(self.charjunk)
924
crqr = cruncher.real_quick_ratio
925
cqr = cruncher.quick_ratio
929
best_i = best_j = None
930
dump_i, dump_j = alo, blo # smallest indices not yet resolved
931
for j in range(blo, bhi):
932
cruncher.set_seq2(b[j])
933
# Search the corresponding i's within WINDOW for rhe highest
934
# ratio greater than `cutoff`.
935
aequiv = alo + (j - blo)
936
arange = range(max(aequiv - WINDOW, dump_i),
937
min(aequiv + WINDOW + 1, ahi))
938
if not arange: # likely exit if `a` is shorter than `b`
942
cruncher.set_seq1(a[i])
943
# Ordering by cheapest to most expensive ratio is very
944
# valuable, most often getting out early.
945
if crqr() <= best_ratio or cqr() <= best_ratio:
949
if ratio > best_ratio:
950
best_i, best_j, best_ratio = i, j, ratio
953
# found nothing to synch on yet - move to next j
956
# pump out straight replace from before this synch pair
957
yield from self._fancy_helper(a, dump_i, best_i,
959
# do intraline marking on the synch pair
960
aelt, belt = a[best_i], b[best_j]
962
# pump out a '-', '?', '+', '?' quad for the synched lines
964
cruncher.set_seqs(aelt, belt)
965
for tag, ai1, ai2, bj1, bj2 in cruncher.get_opcodes():
966
la, lb = ai2 - ai1, bj2 - bj1
970
elif tag == 'delete':
972
elif tag == 'insert':
978
raise ValueError('unknown tag %r' % (tag,))
979
yield from self._qformat(aelt, belt, atags, btags)
981
# the synch pair is identical
983
dump_i, dump_j = best_i + 1, best_j + 1
984
best_i = best_j = None
986
# pump out straight replace from after the last synch pair
987
yield from self._fancy_helper(a, dump_i, ahi,
990
def _fancy_helper(self, a, alo, ahi, b, blo, bhi):
994
g = self._plain_replace(a, alo, ahi, b, blo, bhi)
996
g = self._dump('-', a, alo, ahi)
998
g = self._dump('+', b, blo, bhi)
1002
def _qformat(self, aline, bline, atags, btags):
1004
Format "?" output and deal with tabs.
1009
>>> results = d._qformat('\tabcDefghiJkl\n', '\tabcdefGhijkl\n',
1010
... ' ^ ^ ^ ', ' ^ ^ ^ ')
1011
>>> for line in results: print(repr(line))
1013
'- \tabcDefghiJkl\n'
1015
'+ \tabcdefGhijkl\n'
1018
atags = _keep_original_ws(aline, atags).rstrip()
1019
btags = _keep_original_ws(bline, btags).rstrip()
1023
yield f"? {atags}\n"
1027
yield f"? {btags}\n"
1029
# With respect to junk, an earlier version of ndiff simply refused to
1030
# *start* a match with a junk element. The result was cases like this:
1031
# before: private Thread currentThread;
1032
# after: private volatile Thread currentThread;
1033
# If you consider whitespace to be junk, the longest contiguous match
1034
# not starting with junk is "e Thread currentThread". So ndiff reported
1035
# that "e volatil" was inserted between the 't' and the 'e' in "private".
1036
# While an accurate view, to people that's absurd. The current version
1037
# looks for matching blocks that are entirely junk-free, then extends the
1038
# longest one of those as far as possible but only with matching junk.
1039
# So now "currentThread" is matched, then extended to suck up the
1040
# preceding blank; then "private" is matched, and extended to suck up the
1041
# following blank; then "Thread" is matched; and finally ndiff reports
1042
# that "volatile " was inserted before "Thread". The only quibble
1043
# remaining is that perhaps it was really the case that " volatile"
1044
# was inserted after "private". I can live with that <wink>.
1046
def IS_LINE_JUNK(line, pat=None):
1048
Return True for ignorable line: if `line` is blank or contains a single '#'.
1052
>>> IS_LINE_JUNK('\n')
1054
>>> IS_LINE_JUNK(' # \n')
1056
>>> IS_LINE_JUNK('hello\n')
1061
# Default: match '#' or the empty string
1062
return line.strip() in '#'
1063
# Previous versions used the undocumented parameter 'pat' as a
1064
# match function. Retain this behaviour for compatibility.
1065
return pat(line) is not None
1067
def IS_CHARACTER_JUNK(ch, ws=" \t"):
1069
Return True for ignorable character: iff `ch` is a space or tab.
1073
>>> IS_CHARACTER_JUNK(' ')
1075
>>> IS_CHARACTER_JUNK('\t')
1077
>>> IS_CHARACTER_JUNK('\n')
1079
>>> IS_CHARACTER_JUNK('x')
1086
########################################################################
1088
########################################################################
1090
def _format_range_unified(start, stop):
1091
'Convert range to the "ed" format'
1092
# Per the diff spec at http://www.unix.org/single_unix_specification/
1093
beginning = start + 1 # lines start numbering with one
1094
length = stop - start
1096
return '{}'.format(beginning)
1098
beginning -= 1 # empty ranges begin at line just before the range
1099
return '{},{}'.format(beginning, length)
1101
def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
1102
tofiledate='', n=3, lineterm='\n', *, color=False):
1104
Compare two sequences of lines; generate the delta as a unified diff.
1106
Unified diffs are a compact way of showing line changes and a few
1107
lines of context. The number of context lines is set by 'n' which
1110
By default, the diff control lines (those with ---, +++, or @@) are
1111
created with a trailing newline. This is helpful so that inputs
1112
created from file.readlines() result in diffs that are suitable for
1113
file.writelines() since both the inputs and outputs have trailing
1116
For inputs that do not have trailing newlines, set the lineterm
1117
argument to "" so that the output will be uniformly newline free.
1119
Set 'color' to True to enable output in color, similar to
1120
'git diff --color'. Even if enabled, it can be
1121
controlled using environment variables such as 'NO_COLOR'.
1123
The unidiff format normally has a header for filenames and modification
1124
times. Any or all of these may be specified using strings for
1125
'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'.
1126
The modification times are normally expressed in the ISO 8601 format.
1130
>>> for line in unified_diff('one two three four'.split(),
1131
... 'zero one tree four'.split(), 'Original', 'Current',
1132
... '2005-01-26 23:30:50', '2010-04-02 10:20:52',
1134
... print(line) # doctest: +NORMALIZE_WHITESPACE
1135
--- Original 2005-01-26 23:30:50
1136
+++ Current 2010-04-02 10:20:52
1146
if color and can_colorize():
1147
t = get_theme(force_color=True).difflib
1149
t = get_theme(force_no_color=True).difflib
1151
_check_types(a, b, fromfile, tofile, fromfiledate, tofiledate, lineterm)
1153
for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
1156
fromdate = '\t{}'.format(fromfiledate) if fromfiledate else ''
1157
todate = '\t{}'.format(tofiledate) if tofiledate else ''
1158
yield f'{t.header}--- {fromfile}{fromdate}{lineterm}{t.reset}'
1159
yield f'{t.header}+++ {tofile}{todate}{lineterm}{t.reset}'
1161
first, last = group[0], group[-1]
1162
file1_range = _format_range_unified(first[1], last[2])
1163
file2_range = _format_range_unified(first[3], last[4])
1164
yield f'{t.hunk}@@ -{file1_range} +{file2_range} @@{lineterm}{t.reset}'
1166
for tag, i1, i2, j1, j2 in group:
1168
for line in a[i1:i2]:
1169
yield f'{t.context} {line}{t.reset}'
1171
if tag in {'replace', 'delete'}:
1172
for line in a[i1:i2]:
1173
yield f'{t.removed}-{line}{t.reset}'
1174
if tag in {'replace', 'insert'}:
1175
for line in b[j1:j2]:
1176
yield f'{t.added}+{line}{t.reset}'
1179
########################################################################
1181
########################################################################
1183
def _format_range_context(start, stop):
1184
'Convert range to the "ed" format'
1185
# Per the diff spec at http://www.unix.org/single_unix_specification/
1186
beginning = start + 1 # lines start numbering with one
1187
length = stop - start
1189
beginning -= 1 # empty ranges begin at line just before the range
1191
return '{}'.format(beginning)
1192
return '{},{}'.format(beginning, beginning + length - 1)
1194
# See http://www.unix.org/single_unix_specification/
1195
def context_diff(a, b, fromfile='', tofile='',
1196
fromfiledate='', tofiledate='', n=3, lineterm='\n'):
1198
Compare two sequences of lines; generate the delta as a context diff.
1200
Context diffs are a compact way of showing line changes and a few
1201
lines of context. The number of context lines is set by 'n' which
1204
By default, the diff control lines (those with *** or ---) are
1205
created with a trailing newline. This is helpful so that inputs
1206
created from file.readlines() result in diffs that are suitable for
1207
file.writelines() since both the inputs and outputs have trailing
1210
For inputs that do not have trailing newlines, set the lineterm
1211
argument to "" so that the output will be uniformly newline free.
1213
The context diff format normally has a header for filenames and
1214
modification times. Any or all of these may be specified using
1215
strings for 'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'.
1216
The modification times are normally expressed in the ISO 8601 format.
1217
If not specified, the strings default to blanks.
1221
>>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(True),
1222
... 'zero\none\ntree\nfour\n'.splitlines(True), 'Original', 'Current')),
1239
_check_types(a, b, fromfile, tofile, fromfiledate, tofiledate, lineterm)
1240
prefix = dict(insert='+ ', delete='- ', replace='! ', equal=' ')
1242
for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
1245
fromdate = '\t{}'.format(fromfiledate) if fromfiledate else ''
1246
todate = '\t{}'.format(tofiledate) if tofiledate else ''
1247
yield '*** {}{}{}'.format(fromfile, fromdate, lineterm)
1248
yield '--- {}{}{}'.format(tofile, todate, lineterm)
1250
first, last = group[0], group[-1]
1251
yield '***************' + lineterm
1253
file1_range = _format_range_context(first[1], last[2])
1254
yield '*** {} ****{}'.format(file1_range, lineterm)
1256
if any(tag in {'replace', 'delete'} for tag, _, _, _, _ in group):
1257
for tag, i1, i2, _, _ in group:
1259
for line in a[i1:i2]:
1260
yield prefix[tag] + line
1262
file2_range = _format_range_context(first[3], last[4])
1263
yield '--- {} ----{}'.format(file2_range, lineterm)
1265
if any(tag in {'replace', 'insert'} for tag, _, _, _, _ in group):
1266
for tag, _, _, j1, j2 in group:
1268
for line in b[j1:j2]:
1269
yield prefix[tag] + line
1271
def _check_types(a, b, *args):
1272
# Checking types is weird, but the alternative is garbled output when
1273
# someone passes mixed bytes and str to {unified,context}_diff(). E.g.
1274
# without this check, passing filenames as bytes results in output like
1275
# --- b'oldfile.txt'
1276
# +++ b'newfile.txt'
1277
# because of how str.format() incorporates bytes objects.
1278
if a and not isinstance(a[0], str):
1279
raise TypeError('lines to compare must be str, not %s (%r)' %
1280
(type(a[0]).__name__, a[0]))
1281
if b and not isinstance(b[0], str):
1282
raise TypeError('lines to compare must be str, not %s (%r)' %
1283
(type(b[0]).__name__, b[0]))
1284
if isinstance(a, str):
1285
raise TypeError('input must be a sequence of strings, not %s' %
1287
if isinstance(b, str):
1288
raise TypeError('input must be a sequence of strings, not %s' %
1291
if not isinstance(arg, str):
1292
raise TypeError('all arguments must be str, not: %r' % (arg,))
1294
def diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'',
1295
fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\n'):
1297
Compare `a` and `b`, two sequences of lines represented as bytes rather
1298
than str. This is a wrapper for `dfunc`, which is typically either
1299
unified_diff() or context_diff(). Inputs are losslessly converted to
1300
strings so that `dfunc` only has to worry about strings, and encoded
1301
back to bytes on return. This is necessary to compare files with
1302
unknown or inconsistent encoding. All other inputs (except `n`) must be
1303
bytes rather than str.
1307
return s.decode('ascii', 'surrogateescape')
1308
except AttributeError as err:
1309
msg = ('all arguments must be bytes, not %s (%r)' %
1310
(type(s).__name__, s))
1311
raise TypeError(msg) from err
1312
a = list(map(decode, a))
1313
b = list(map(decode, b))
1314
fromfile = decode(fromfile)
1315
tofile = decode(tofile)
1316
fromfiledate = decode(fromfiledate)
1317
tofiledate = decode(tofiledate)
1318
lineterm = decode(lineterm)
1320
lines = dfunc(a, b, fromfile, tofile, fromfiledate, tofiledate, n, lineterm)
1322
yield line.encode('ascii', 'surrogateescape')
1324
def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
1326
Compare `a` and `b` (lists of strings); return a `Differ`-style delta.
1328
Optional keyword parameters `linejunk` and `charjunk` are for filter
1329
functions, or can be None:
1331
- linejunk: A function that should accept a single string argument and
1332
return true iff the string is junk. The default is None, and is
1333
recommended; the underlying SequenceMatcher class has an adaptive
1334
notion of "noise" lines.
1336
- charjunk: A function that accepts a character (string of length
1337
1), and returns true iff the character is junk. The default is
1338
the module-level function IS_CHARACTER_JUNK, which filters out
1339
whitespace characters (a blank or tab; note: it's a bad idea to
1340
include newline in this!).
1342
Tools/scripts/ndiff.py is a command-line front-end to this function.
1346
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
1347
... 'ore\ntree\nemu\n'.splitlines(keepends=True))
1348
>>> print(''.join(diff), end="")
1359
return Differ(linejunk, charjunk).compare(a, b)
1361
def _mdiff(fromlines, tolines, context=None, linejunk=None,
1362
charjunk=IS_CHARACTER_JUNK):
1363
r"""Returns generator yielding marked up from/to side by side differences.
1366
fromlines -- list of text lines to compared to tolines
1367
tolines -- list of text lines to be compared to fromlines
1368
context -- number of context lines to display on each side of difference,
1369
if None, all from/to text lines will be generated.
1370
linejunk -- passed on to ndiff (see ndiff documentation)
1371
charjunk -- passed on to ndiff (see ndiff documentation)
1373
This function returns an iterator which returns a tuple:
1374
(from line tuple, to line tuple, boolean flag)
1376
from/to line tuple -- (line num, line text)
1377
line num -- integer or None (to indicate a context separation)
1378
line text -- original line text with following markers inserted:
1379
'\0+' -- marks start of added text
1380
'\0-' -- marks start of deleted text
1381
'\0^' -- marks start of changed text
1382
'\1' -- marks end of added/deleted/changed text
1384
boolean flag -- None indicates context separation, True indicates
1385
either "from" or "to" line contains a change, otherwise False.
1387
This function/iterator was originally developed to generate side by side
1388
file difference for making HTML pages (see HtmlDiff class for example
1391
Note, this function utilizes the ndiff function to generate the side by
1392
side difference markup. Optional ndiff arguments may be passed to this
1393
function and they in turn will be passed to ndiff.
1397
# regular expression for finding intraline change indices
1398
change_re = re.compile(r'(\++|\-+|\^+)')
1400
# create the difference iterator to generate the differences
1401
diff_lines_iterator = ndiff(fromlines,tolines,linejunk,charjunk)
1403
def _make_line(lines, format_key, side, num_lines=[0,0]):
1404
"""Returns line of text with user's change markup and line formatting.
1406
lines -- list of lines from the ndiff generator to produce a line of
1407
text from. When producing the line of text to return, the
1408
lines used are removed from this list.
1409
format_key -- '+' return first line in list with "add" markup around
1411
'-' return first line in list with "delete" markup around
1413
'?' return first line in list with add/delete/change
1414
intraline markup (indices obtained from second line)
1415
None return first line in list with no markup
1416
side -- indice into the num_lines list (0=from,1=to)
1417
num_lines -- from/to current line number. This is NOT intended to be a
1418
passed parameter. It is present as a keyword argument to
1419
maintain memory of the current line numbers between calls
1422
Note, this function is purposefully not defined at the module scope so
1423
that data it needs from its parent function (within whose context it
1424
is defined) does not need to be of module scope.
1426
num_lines[side] += 1
1427
# Handle case where no user markup is to be added, just return line of
1428
# text with user's line format to allow for usage of the line number.
1429
if format_key is None:
1430
return (num_lines[side],lines.pop(0)[2:])
1431
# Handle case of intraline changes
1432
if format_key == '?':
1433
text, markers = lines.pop(0), lines.pop(0)
1434
# find intraline changes (store change type and indices in tuples)
1436
def record_sub_info(match_object,sub_info=sub_info):
1437
sub_info.append([match_object.group(1)[0],match_object.span()])
1438
return match_object.group(1)
1439
change_re.sub(record_sub_info,markers)
1440
# process each tuple inserting our special marks that won't be
1441
# noticed by an xml/html escaper.
1442
for key,(begin,end) in reversed(sub_info):
1443
text = text[0:begin]+'\0'+key+text[begin:end]+'\1'+text[end:]
1445
# Handle case of add/delete entire line
1447
text = lines.pop(0)[2:]
1448
# if line of text is just a newline, insert a space so there is
1449
# something for the user to highlight and see.
1452
# insert marks that won't be noticed by an xml/html escaper.
1453
text = '\0' + format_key + text + '\1'
1454
# Return line of text, first allow user's line formatter to do its
1455
# thing (such as adding the line number) then replace the special
1456
# marks with what the user's change markup.
1457
return (num_lines[side],text)
1459
def _line_iterator():
1460
"""Yields from/to lines of text with a change indication.
1462
This function is an iterator. It itself pulls lines from a
1463
differencing iterator, processes them and yields them. When it can
1464
it yields both a "from" and a "to" line, otherwise it will yield one
1465
or the other. In addition to yielding the lines of from/to text, a
1466
boolean flag is yielded to indicate if the text line(s) have
1467
differences in them.
1469
Note, this function is purposefully not defined at the module scope so
1470
that data it needs from its parent function (within whose context it
1471
is defined) does not need to be of module scope.
1474
num_blanks_pending, num_blanks_to_yield = 0, 0
1476
# Load up next 4 lines so we can look ahead, create strings which
1477
# are a concatenation of the first character of each of the 4 lines
1478
# so we can do some very readable comparisons.
1479
while len(lines) < 4:
1480
lines.append(next(diff_lines_iterator, 'X'))
1481
s = ''.join([line[0] for line in lines])
1482
if s.startswith('X'):
1483
# When no more lines, pump out any remaining blank lines so the
1484
# corresponding add/delete lines get a matching blank line so
1485
# all line pairs get yielded at the next level.
1486
num_blanks_to_yield = num_blanks_pending
1487
elif s.startswith('-?+?'):
1488
# simple intraline change
1489
yield _make_line(lines,'?',0), _make_line(lines,'?',1), True
1491
elif s.startswith('--++'):
1492
# in delete block, add block coming: we do NOT want to get
1493
# caught up on blank lines yet, just process the delete line
1494
num_blanks_pending -= 1
1495
yield _make_line(lines,'-',0), None, True
1497
elif s.startswith(('--?+', '--+', '- ')):
1498
# in delete block and see an intraline change or unchanged line
1499
# coming: yield the delete line and then blanks
1500
from_line,to_line = _make_line(lines,'-',0), None
1501
num_blanks_to_yield,num_blanks_pending = num_blanks_pending-1,0
1502
elif s.startswith('-+?'):
1504
yield _make_line(lines,None,0), _make_line(lines,'?',1), True
1506
elif s.startswith('-?+'):
1508
yield _make_line(lines,'?',0), _make_line(lines,None,1), True
1510
elif s.startswith('-'):
1512
num_blanks_pending -= 1
1513
yield _make_line(lines,'-',0), None, True
1515
elif s.startswith('+--'):
1516
# in add block, delete block coming: we do NOT want to get
1517
# caught up on blank lines yet, just process the add line
1518
num_blanks_pending += 1
1519
yield None, _make_line(lines,'+',1), True
1521
elif s.startswith(('+ ', '+-')):
1522
# will be leaving an add block: yield blanks then add line
1523
from_line, to_line = None, _make_line(lines,'+',1)
1524
num_blanks_to_yield,num_blanks_pending = num_blanks_pending+1,0
1525
elif s.startswith('+'):
1526
# inside an add block, yield the add line
1527
num_blanks_pending += 1
1528
yield None, _make_line(lines,'+',1), True
1530
elif s.startswith(' '):
1531
# unchanged text, yield it to both sides
1532
yield _make_line(lines[:],None,0),_make_line(lines,None,1),False
1534
# Catch up on the blank lines so when we yield the next from/to
1535
# pair, they are lined up.
1536
while(num_blanks_to_yield < 0):
1537
num_blanks_to_yield += 1
1538
yield None,('','\n'),True
1539
while(num_blanks_to_yield > 0):
1540
num_blanks_to_yield -= 1
1541
yield ('','\n'),None,True
1542
if s.startswith('X'):
1545
yield from_line,to_line,True
1547
def _line_pair_iterator():
1548
"""Yields from/to lines of text with a change indication.
1550
This function is an iterator. It itself pulls lines from the line
1551
iterator. Its difference from that iterator is that this function
1552
always yields a pair of from/to text lines (with the change
1553
indication). If necessary it will collect single from/to lines
1554
until it has a matching pair from/to pair to yield.
1556
Note, this function is purposefully not defined at the module scope so
1557
that data it needs from its parent function (within whose context it
1558
is defined) does not need to be of module scope.
1560
line_iterator = _line_iterator()
1561
fromlines,tolines=[],[]
1563
# Collecting lines of text until we have a from/to pair
1564
while (len(fromlines)==0 or len(tolines)==0):
1566
from_line, to_line, found_diff = next(line_iterator)
1567
except StopIteration:
1569
if from_line is not None:
1570
fromlines.append((from_line,found_diff))
1571
if to_line is not None:
1572
tolines.append((to_line,found_diff))
1573
# Once we have a pair, remove them from the collection and yield it
1574
from_line, fromDiff = fromlines.pop(0)
1575
to_line, to_diff = tolines.pop(0)
1576
yield (from_line,to_line,fromDiff or to_diff)
1578
# Handle case where user does not want context differencing, just yield
1579
# them up without doing anything else with them.
1580
line_pair_iterator = _line_pair_iterator()
1582
yield from line_pair_iterator
1583
# Handle case where user wants context differencing. We must do some
1584
# storage of lines until we know for sure that they are to be yielded.
1589
# Store lines up until we find a difference, note use of a
1590
# circular queue because we only need to keep around what
1591
# we need for context.
1592
index, contextLines = 0, [None]*(context)
1594
while(found_diff is False):
1596
from_line, to_line, found_diff = next(line_pair_iterator)
1597
except StopIteration:
1600
contextLines[i] = (from_line, to_line, found_diff)
1602
# Yield lines that we have collected so far, but first yield
1603
# the user's separator.
1605
yield None, None, None
1606
lines_to_write = context
1608
lines_to_write = index
1610
while(lines_to_write):
1613
yield contextLines[i]
1615
# Now yield the context lines after the change
1616
lines_to_write = context-1
1618
while(lines_to_write):
1619
from_line, to_line, found_diff = next(line_pair_iterator)
1620
# If another change within the context, extend the context
1622
lines_to_write = context-1
1625
yield from_line, to_line, found_diff
1626
except StopIteration:
1627
# Catch exception from next() and return normally
1631
_file_template = """
1635
<meta charset="%(charset)s">
1636
<meta name="viewport" content="width=device-width, initial-scale=1">
1637
<title>Diff comparison</title>
1649
:root {color-scheme: light dark}
1651
font-family: Menlo, Consolas, Monaco, Liberation Mono, Lucida Console, monospace;
1655
background-color: #e0e0e0;
1663
background-color: #c0c0c0;
1666
.diff_add {background-color:palegreen}
1667
.diff_chg {background-color:#ffff77}
1668
.diff_sub {background-color:#ffaaaa}
1669
table.diff[summary="Legends"] {
1671
border: 1px solid #ccc;
1673
table.diff[summary="Legends"] th {
1674
background-color: #e0e0e0;
1677
table.diff[summary="Legends"] td {
1681
@media (prefers-color-scheme: dark) {
1682
.diff_header {background-color:#666}
1683
.diff_next {background-color:#393939}
1684
.diff_add {background-color:darkgreen}
1685
.diff_chg {background-color:#847415}
1686
.diff_sub {background-color:darkred}
1687
table.diff[summary="Legends"] {border-color:#555}
1688
table.diff[summary="Legends"] th{background-color:#666}
1691
_table_template = """
1692
<table class="diff" id="difflib_chg_%(prefix)s_top"
1693
cellspacing="0" cellpadding="0" rules="groups" >
1694
<colgroup></colgroup> <colgroup></colgroup> <colgroup></colgroup>
1695
<colgroup></colgroup> <colgroup></colgroup> <colgroup></colgroup>
1698
%(data_rows)s </tbody>
1702
<table class="diff" summary="Legends">
1703
<tr> <th colspan="2"> Legends </th> </tr>
1704
<tr> <td> <table border="" summary="Colors">
1705
<tr><th> Colors </th> </tr>
1706
<tr><td class="diff_add"> Added </td></tr>
1707
<tr><td class="diff_chg">Changed</td> </tr>
1708
<tr><td class="diff_sub">Deleted</td> </tr>
1710
<td> <table border="" summary="Links">
1711
<tr><th colspan="2"> Links </th> </tr>
1712
<tr><td>(f)irst change</td> </tr>
1713
<tr><td>(n)ext change</td> </tr>
1714
<tr><td>(t)op</td> </tr>
1718
class HtmlDiff(object):
1719
"""For producing HTML side by side comparison with change highlights.
1721
This class can be used to create an HTML table (or a complete HTML file
1722
containing the table) showing a side by side, line by line comparison
1723
of text with inter-line and intra-line change highlights. The table can
1724
be generated in either full or contextual difference mode.
1726
The following methods are provided for HTML generation:
1728
make_table -- generates HTML for a single side by side table
1729
make_file -- generates complete HTML file with a single side by side table
1731
See Doc/includes/diff.py for an example usage of this class.
1734
_file_template = _file_template
1736
_table_template = _table_template
1740
def __init__(self,tabsize=8,wrapcolumn=None,linejunk=None,
1741
charjunk=IS_CHARACTER_JUNK):
1742
"""HtmlDiff instance initializer
1745
tabsize -- tab stop spacing, defaults to 8.
1746
wrapcolumn -- column number where lines are broken and wrapped,
1747
defaults to None where lines are not wrapped.
1748
linejunk,charjunk -- keyword arguments passed into ndiff() (used by
1749
HtmlDiff() to generate the side by side HTML differences). See
1750
ndiff() documentation for argument default values and descriptions.
1752
self._tabsize = tabsize
1753
self._wrapcolumn = wrapcolumn
1754
self._linejunk = linejunk
1755
self._charjunk = charjunk
1757
def make_file(self, fromlines, tolines, fromdesc='', todesc='',
1758
context=False, numlines=5, *, charset='utf-8'):
1759
"""Returns HTML file of side by side comparison with change highlights
1762
fromlines -- list of "from" lines
1763
tolines -- list of "to" lines
1764
fromdesc -- "from" file column header string
1765
todesc -- "to" file column header string
1766
context -- set to True for contextual differences (defaults to False
1767
which shows full differences).
1768
numlines -- number of context lines. When context is set True,
1769
controls number of lines displayed before and after the change.
1770
When context is False, controls the number of lines to place
1771
the "next" link anchors before the next change (so click of
1772
"next" link jumps to just before the change).
1773
charset -- charset of the HTML document
1776
return (self._file_template % dict(
1777
styles=self._styles,
1778
legend=self._legend,
1779
table=self.make_table(fromlines, tolines, fromdesc, todesc,
1780
context=context, numlines=numlines),
1782
)).encode(charset, 'xmlcharrefreplace').decode(charset)
1784
def _tab_newline_replace(self,fromlines,tolines):
1785
"""Returns from/to line lists with tabs expanded and newlines removed.
1787
Instead of tab characters being replaced by the number of spaces
1788
needed to fill in to the next tab stop, this function will fill
1789
the space with tab characters. This is done so that the difference
1790
algorithms can identify changes in a file when tabs are replaced by
1791
spaces and vice versa. At the end of the HTML generation, the tab
1792
characters will be replaced with a nonbreakable space.
1794
def expand_tabs(line):
1796
line = line.replace(' ','\0')
1797
# expand tabs into spaces
1798
line = line.expandtabs(self._tabsize)
1799
# replace spaces from expanded tabs back into tab characters
1800
# (we'll replace them with markup after we do differencing)
1801
line = line.replace(' ','\t')
1802
return line.replace('\0',' ').rstrip('\n')
1803
fromlines = [expand_tabs(line) for line in fromlines]
1804
tolines = [expand_tabs(line) for line in tolines]
1805
return fromlines,tolines
1807
def _split_line(self,data_list,line_num,text):
1808
"""Builds list of text lines by splitting text lines at wrap point
1810
This function will determine if the input text line needs to be
1811
wrapped (split) into separate lines. If so, the first wrap point
1812
will be determined and the first line appended to the output
1813
text line list. This function is used recursively to handle
1814
the second part of the split line to further split it.
1816
# if blank line or context separator, just add it to the output list
1818
data_list.append((line_num,text))
1821
# if line text doesn't need wrapping, just add it to the output list
1823
max = self._wrapcolumn
1824
if (size <= max) or ((size -(text.count('\0')*3)) <= max):
1825
data_list.append((line_num,text))
1828
# scan text looking for the wrap point, keeping track if the wrap
1829
# point is inside markers
1833
while n < max and i < size:
1838
elif text[i] == '\1':
1845
# wrap point is inside text, break it up into separate lines
1849
# if wrap point is inside markers, place end marker at end of first
1850
# line and start marker at beginning of second line because each
1851
# line will have its own table tag markup around it.
1853
line1 = line1 + '\1'
1854
line2 = '\0' + mark + line2
1856
# tack on first line onto the output list
1857
data_list.append((line_num,line1))
1859
# use this routine again to wrap the remaining text
1860
self._split_line(data_list,'>',line2)
1862
def _line_wrapper(self,diffs):
1863
"""Returns iterator that splits (wraps) mdiff text lines"""
1865
# pull from/to data and flags from mdiff iterator
1866
for fromdata,todata,flag in diffs:
1867
# check for context separators and pass them through
1869
yield fromdata,todata,flag
1871
(fromline,fromtext),(toline,totext) = fromdata,todata
1872
# for each from/to line split it at the wrap column to form
1873
# list of text lines.
1874
fromlist,tolist = [],[]
1875
self._split_line(fromlist,fromline,fromtext)
1876
self._split_line(tolist,toline,totext)
1877
# yield from/to line in pairs inserting blank lines as
1878
# necessary when one side has more wrapped lines
1879
while fromlist or tolist:
1881
fromdata = fromlist.pop(0)
1885
todata = tolist.pop(0)
1888
yield fromdata,todata,flag
1890
def _collect_lines(self,diffs):
1891
"""Collects mdiff output into separate lists
1893
Before storing the mdiff from/to data into a list, it is converted
1894
into a single line of text with HTML markup.
1897
fromlist,tolist,flaglist = [],[],[]
1898
# pull from/to data and flags from mdiff style iterator
1899
for fromdata,todata,flag in diffs:
1901
# store HTML markup of the lines into the lists
1902
fromlist.append(self._format_line(0,flag,*fromdata))
1903
tolist.append(self._format_line(1,flag,*todata))
1905
# exceptions occur for lines where context separators go
1906
fromlist.append(None)
1908
flaglist.append(flag)
1909
return fromlist,tolist,flaglist
1911
def _format_line(self,side,flag,linenum,text):
1912
"""Returns HTML markup of "from" / "to" text lines
1914
side -- 0 or 1 indicating "from" or "to" text
1915
flag -- indicates if difference on line
1916
linenum -- line number (used for line number column)
1917
text -- line text to be marked up
1920
linenum = '%d' % linenum
1921
id = ' id="%s%s"' % (self._prefix[side],linenum)
1923
# handle blank lines where linenum is '>' or ''
1925
# replace those things that would get confused with HTML symbols
1926
text=text.replace("&","&").replace(">",">").replace("<","<")
1928
# make space non-breakable so they don't get compressed or line wrapped
1929
text = text.replace(' ',' ').rstrip()
1931
# add a class to the td tag if there is a difference on the line
1932
css_class = ' class="diff_changed" ' if flag else ' '
1934
return f'<td class="diff_header"{id}>{linenum}</td>' \
1935
+ f'<td{css_class}nowrap="nowrap">{text}</td>'
1937
def _make_prefix(self):
1938
"""Create unique anchor prefixes"""
1940
# Generate a unique anchor prefix so multiple tables
1941
# can exist on the same HTML page without conflicts.
1942
fromprefix = "from%d_" % HtmlDiff._default_prefix
1943
toprefix = "to%d_" % HtmlDiff._default_prefix
1944
HtmlDiff._default_prefix += 1
1945
# store prefixes so line format method has access
1946
self._prefix = [fromprefix,toprefix]
1948
def _convert_flags(self,fromlist,tolist,flaglist,context,numlines):
1949
"""Makes list of "next" links"""
1951
# all anchor names will be generated using the unique "to" prefix
1952
toprefix = self._prefix[1]
1954
# process change flags, generating middle column of next anchors/links
1955
next_id = ['']*len(flaglist)
1956
next_href = ['']*len(flaglist)
1957
num_chg, in_change = 0, False
1959
for i,flag in enumerate(flaglist):
1964
# at the beginning of a change, drop an anchor a few lines
1965
# (the context lines) before the change for the previous
1967
i = max([0,i-numlines])
1968
next_id[i] = ' id="difflib_chg_%s_%d"' % (toprefix,num_chg)
1969
# at the beginning of a change, drop a link to the next
1972
next_href[last] = '<a href="#difflib_chg_%s_%d">n</a>' % (
1976
# check for cases where there is no content to avoid exceptions
1983
fromlist = ['<td></td><td> No Differences Found </td>']
1986
fromlist = tolist = ['<td></td><td> Empty File </td>']
1987
# if not a change on first line, drop a link
1989
next_href[0] = '<a href="#difflib_chg_%s_0">f</a>' % toprefix
1990
# redo the last link to link to the top
1991
next_href[last] = '<a href="#difflib_chg_%s_top">t</a>' % (toprefix)
1993
return fromlist,tolist,flaglist,next_href,next_id
1995
def make_table(self,fromlines,tolines,fromdesc='',todesc='',context=False,
1997
"""Returns HTML table of side by side comparison with change highlights
2000
fromlines -- list of "from" lines
2001
tolines -- list of "to" lines
2002
fromdesc -- "from" file column header string
2003
todesc -- "to" file column header string
2004
context -- set to True for contextual differences (defaults to False
2005
which shows full differences).
2006
numlines -- number of context lines. When context is set True,
2007
controls number of lines displayed before and after the change.
2008
When context is False, controls the number of lines to place
2009
the "next" link anchors before the next change (so click of
2010
"next" link jumps to just before the change).
2013
# make unique anchor prefixes so that multiple tables may exist
2014
# on the same page without conflict.
2017
# change tabs to spaces before it gets more difficult after we insert
2019
# it also removes trailing newlines, causing some diffs to be missed
2021
fromlines,tolines = self._tab_newline_replace(fromlines,tolines)
2023
# create diffs iterator which generates side by side from/to data
2025
context_lines = numlines
2027
context_lines = None
2028
diffs = _mdiff(fromlines,tolines,context_lines,linejunk=self._linejunk,
2029
charjunk=self._charjunk)
2031
# set up iterator to wrap lines that exceed desired width
2032
if self._wrapcolumn:
2033
diffs = self._line_wrapper(diffs)
2035
# collect up from/to lines and flags into lists (also format the lines)
2036
fromlist,tolist,flaglist = self._collect_lines(diffs)
2038
# process change flags, generating middle column of next anchors/links
2039
fromlist,tolist,flaglist,next_href,next_id = self._convert_flags(
2040
fromlist,tolist,flaglist,context,numlines)
2043
fmt = ' <tr><td class="diff_next"%s>%s</td>%s' + \
2044
'<td class="diff_next">%s</td>%s</tr>\n'
2045
for i in range(len(flaglist)):
2046
if flaglist[i] is None:
2047
# mdiff yields None on separator lines skip the bogus ones
2048
# generated for the first line
2050
s.append(' </tbody> \n <tbody>\n')
2052
s.append( fmt % (next_id[i],next_href[i],fromlist[i],
2053
next_href[i],tolist[i]))
2054
if fromdesc or todesc:
2055
header_row = '<thead><tr>%s%s%s%s</tr></thead>' % (
2056
'<th class="diff_next"><br /></th>',
2057
'<th colspan="2" class="diff_header">%s</th>' % fromdesc,
2058
'<th class="diff_next"><br /></th>',
2059
'<th colspan="2" class="diff_header">%s</th>' % todesc)
2063
table = self._table_template % dict(
2064
data_rows=''.join(s),
2065
header_row=header_row,
2066
prefix=self._prefix[1])
2068
return table.replace('\0+','<span class="diff_add">'). \
2069
replace('\0-','<span class="diff_sub">'). \
2070
replace('\0^','<span class="diff_chg">'). \
2071
replace('\1','</span>'). \
2072
replace('\t',' ')
2075
def restore(delta, which):
2077
Generate one of the two sequences that generated a delta.
2079
Given a `delta` produced by `Differ.compare()` or `ndiff()`, extract
2080
lines originating from file 1 or 2 (parameter `which`), stripping off line
2085
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
2086
... 'ore\ntree\nemu\n'.splitlines(keepends=True))
2087
>>> diff = list(diff)
2088
>>> print(''.join(restore(diff, 1)), end="")
2092
>>> print(''.join(restore(diff, 2)), end="")
2098
tag = {1: "- ", 2: "+ "}[int(which)]
2100
raise ValueError('unknown delta choice (must be 1 or 2): %r'
2102
prefixes = (" ", tag)
2104
if line[:2] in prefixes: