Correct Re-computation of Relative Addresses in Defined Names (#3673)

* Correct Re-computation of Relative Addresses in Defined Names

Fix #3661. Insertion or deletion of rows or columns can cause changes to the ranges for Defined Names. In fact, only the absolute parts of such ranges should be adjusted, while the relative parts should be left alone. Otherwise, as the original issue documents, the adjustment to the relative portion winds up being double-counted when the Defined Name is referenced in a formula. The major part of this change is to ReferenceHelper and CellReferenceHelper to not adjust relative addresses for Defined Names. An additional small change is needed in the Calculation engine to `recursiveCalculationCell` when a Defined Formula is being calculated.

In a sense, this is a breaking change, but for an obscure use case which (a) was wrong, and (b) is unlikely to be of importance. Some of the tests in ReferenceHelperTest were wrong and are now corrected, with the results being cross-checked against Excel.

When a Defined Name using relative addressing is defined in Excel, the result is treated as relative to the active cell on the sheet in which the name is defined. PhpSpreadsheet treats it as relative to cell A1. I think that is a reasonable treatment, and will not change its behavior to match Excel's - that would definitely be a breaking change of some consequence.

An interesting use of relative address in a defined name is demonstrated at https://excelguru.ca/always-refer-to-the-cell-above/. Note that the steps there involve setting the selected cell to A2 before defining the name. When that spreadsheet is stored, the actual definition of the range is `A1048576`. Likewise, adding a defined name for the cell to the left would be stored as `XFD1`. This seems a little fragile, but Ods, which I believe does not have the same row and column limits as Excel, certainly treats these values the same as Excel. This particular construction is formally unit-tested. Note, however, that although using these Defined Names as a formula on their own works just fine, a construction like `=SUM(A1:CellAbove)`, as suggested in the article, seems to put PhpSpreadsheet calculation engine in a loop. In the likely event that I can't solve that before I merge this change, I will open a new issue to that effect when I do merge it. Note that this can be handled without defined names as `=SUM(A$1:INDIRECT(ADDRESS(ROW()-1,COLUMN())))`. PhpSpreadsheet will handle this as a cell formula, but not yet as a Named Formula.

The tests show a breakdown evaluating `=ProductTotal` (product of 2 formulas using defined names with relative addresses) on the sheet on which it is defined, but it works from a different sheet. The usual debugging techniques show me why this is happening, but I can't see how to overcome it. As above, if I can't solve it before I merge, I will open a new issue.

For those situations where I intend to open a new issue, tests are added but are marked Incomplete. Because of those, I will leave this PR in draft status for 2 weeks before moving forward with it.

* Fix productTotal Problem

Need to restore current cell after evaluating defined name.

Author: oleibman

src/PhpSpreadsheet/Calculation/Calculation.php

@@ -5745,7 +5745,8 @@ private function evaluateDefinedName(Cell $cell, DefinedName $namedRange, Worksh
 
         $this->debugLog->writeDebugLog('Defined Name is a %s with a value of %s', $definedNameType, $definedNameValue);
 
-        $recursiveCalculationCell = ($definedNameWorksheet !== null && $definedNameWorksheet !== $cellWorksheet)
+        $originalCoordinate = $cell->getCoordinate();
+        $recursiveCalculationCell = ($definedNameType !== 'Formula' && $definedNameWorksheet !== null && $definedNameWorksheet !== $cellWorksheet)
             ? $definedNameWorksheet->getCell('A1')
             : $cell;
         $recursiveCalculationCellAddress = $recursiveCalculationCell->getCoordinate();
@@ -5763,6 +5764,7 @@ private function evaluateDefinedName(Cell $cell, DefinedName $namedRange, Worksh
         $recursiveCalculator->getDebugLog()->setWriteDebugLog($this->getDebugLog()->getWriteDebugLog());
         $recursiveCalculator->getDebugLog()->setEchoDebugLog($this->getDebugLog()->getEchoDebugLog());
         $result = $recursiveCalculator->_calculateFormulaValue($definedNameValue, $recursiveCalculationCellAddress, $recursiveCalculationCell, true);
+        $cellWorksheet->getCell($originalCoordinate);
 
         if ($this->getDebugLog()->getWriteDebugLog()) {
             $this->debugLog->mergeDebugLog(array_slice($recursiveCalculator->getDebugLog()->getLog(), 3));

src/PhpSpreadsheet/CellReferenceHelper.php

@@ -56,7 +56,7 @@ public function refreshRequired(string $beforeCellAddress, int $numberOfColumns,
             $this->numberOfRows !== $numberOfRows;
     }
 
-    public function updateCellReference(string $cellReference = 'A1', bool $includeAbsoluteReferences = false): string
+    public function updateCellReference(string $cellReference = 'A1', bool $includeAbsoluteReferences = false, bool $onlyAbsoluteReferences = false): string
     {
         if (Coordinate::coordinateIsRange($cellReference)) {
             throw new Exception('Only single cell references may be passed to this method.');
@@ -70,7 +70,10 @@ public function updateCellReference(string $cellReference = 'A1', bool $includeA
         $absoluteColumn = $newColumn[0] === '$' ? '$' : '';
         $absoluteRow = $newRow[0] === '$' ? '$' : '';
         // Verify which parts should be updated
-        if ($includeAbsoluteReferences === false) {
+        if ($onlyAbsoluteReferences === true) {
+            $updateColumn = (($absoluteColumn === '$') && $newColumnIndex >= $this->beforeColumn);
+            $updateRow = (($absoluteRow === '$') && $newRowIndex >= $this->beforeRow);
+        } elseif ($includeAbsoluteReferences === false) {
             $updateColumn = (($absoluteColumn !== '$') && $newColumnIndex >= $this->beforeColumn);
             $updateRow = (($absoluteRow !== '$') && $newRowIndex >= $this->beforeRow);
         } else {

src/PhpSpreadsheet/ReferenceHelper.php

@@ -3,6 +3,7 @@
 namespace PhpOffice\PhpSpreadsheet;
 
 use PhpOffice\PhpSpreadsheet\Calculation\Calculation;
+use PhpOffice\PhpSpreadsheet\Cell\AddressRange;
 use PhpOffice\PhpSpreadsheet\Cell\Coordinate;
 use PhpOffice\PhpSpreadsheet\Cell\DataType;
 use PhpOffice\PhpSpreadsheet\Style\Conditional;
@@ -572,7 +573,8 @@ public function updateFormulaReferences(
         $numberOfColumns = 0,
         $numberOfRows = 0,
         $worksheetName = '',
-        bool $includeAbsoluteReferences = false
+        bool $includeAbsoluteReferences = false,
+        bool $onlyAbsoluteReferences = false
     ) {
         if (
             $this->cellReferenceHelper === null ||
@@ -596,8 +598,8 @@ public function updateFormulaReferences(
                     foreach ($matches as $match) {
                         $fromString = ($match[2] > '') ? $match[2] . '!' : '';
                         $fromString .= $match[3] . ':' . $match[4];
-                        $modified3 = substr($this->updateCellReference('$A' . $match[3], $includeAbsoluteReferences), 2);
-                        $modified4 = substr($this->updateCellReference('$A' . $match[4], $includeAbsoluteReferences), 2);
+                        $modified3 = substr($this->updateCellReference('$A' . $match[3], $includeAbsoluteReferences, $onlyAbsoluteReferences), 2);
+                        $modified4 = substr($this->updateCellReference('$A' . $match[4], $includeAbsoluteReferences, $onlyAbsoluteReferences), 2);
 
                         if ($match[3] . ':' . $match[4] !== $modified3 . ':' . $modified4) {
                             if (($match[2] == '') || (trim($match[2], "'") == $worksheetName)) {
@@ -621,8 +623,8 @@ public function updateFormulaReferences(
                     foreach ($matches as $match) {
                         $fromString = ($match[2] > '') ? $match[2] . '!' : '';
                         $fromString .= $match[3] . ':' . $match[4];
-                        $modified3 = substr($this->updateCellReference($match[3] . '$1', $includeAbsoluteReferences), 0, -2);
-                        $modified4 = substr($this->updateCellReference($match[4] . '$1', $includeAbsoluteReferences), 0, -2);
+                        $modified3 = substr($this->updateCellReference($match[3] . '$1', $includeAbsoluteReferences, $onlyAbsoluteReferences), 0, -2);
+                        $modified4 = substr($this->updateCellReference($match[4] . '$1', $includeAbsoluteReferences, $onlyAbsoluteReferences), 0, -2);
 
                         if ($match[3] . ':' . $match[4] !== $modified3 . ':' . $modified4) {
                             if (($match[2] == '') || (trim($match[2], "'") == $worksheetName)) {
@@ -646,8 +648,8 @@ public function updateFormulaReferences(
                     foreach ($matches as $match) {
                         $fromString = ($match[2] > '') ? $match[2] . '!' : '';
                         $fromString .= $match[3] . ':' . $match[4];
-                        $modified3 = $this->updateCellReference($match[3], $includeAbsoluteReferences);
-                        $modified4 = $this->updateCellReference($match[4], $includeAbsoluteReferences);
+                        $modified3 = $this->updateCellReference($match[3], $includeAbsoluteReferences, $onlyAbsoluteReferences);
+                        $modified4 = $this->updateCellReference($match[4], $includeAbsoluteReferences, $onlyAbsoluteReferences);
 
                         if ($match[3] . $match[4] !== $modified3 . $modified4) {
                             if (($match[2] == '') || (trim($match[2], "'") == $worksheetName)) {
@@ -674,7 +676,7 @@ public function updateFormulaReferences(
                         $fromString = ($match[2] > '') ? $match[2] . '!' : '';
                         $fromString .= $match[3];
 
-                        $modified3 = $this->updateCellReference($match[3], $includeAbsoluteReferences);
+                        $modified3 = $this->updateCellReference($match[3], $includeAbsoluteReferences, $onlyAbsoluteReferences);
                         if ($match[3] !== $modified3) {
                             if (($match[2] == '') || (trim($match[2], "'") == $worksheetName)) {
                                 $toString = ($match[2] > '') ? $match[2] . '!' : '';
@@ -757,11 +759,13 @@ private function updateCellReferencesAllWorksheets(string $formula, int $numberO
             $row = $rows[$splitCount][0];
 
             if (!empty($column) && $column[0] !== '$') {
-                $column = Coordinate::stringFromColumnIndex(Coordinate::columnIndexFromString($column) + $numberOfColumns);
+                $column = ((Coordinate::columnIndexFromString($column) + $numberOfColumns) % AddressRange::MAX_COLUMN_INT) ?: AddressRange::MAX_COLUMN_INT;
+                $column = Coordinate::stringFromColumnIndex($column);
+                $rowOffset -= ($columnLength - strlen($column));
                 $formula = substr($formula, 0, $columnOffset) . $column . substr($formula, $columnOffset + $columnLength);
             }
             if (!empty($row) && $row[0] !== '$') {
-                $row = (int) $row + $numberOfRows;
+                $row = (((int) $row + $numberOfRows) % AddressRange::MAX_ROW) ?: AddressRange::MAX_ROW;
                 $formula = substr($formula, 0, $rowOffset) . $row . substr($formula, $rowOffset + $rowLength);
             }
         }
@@ -854,7 +858,7 @@ private function updateRowRangesAllWorksheets(string $formula, int $numberOfRows
      *
      * @return string Updated cell range
      */
-    private function updateCellReference($cellReference = 'A1', bool $includeAbsoluteReferences = false)
+    private function updateCellReference($cellReference = 'A1', bool $includeAbsoluteReferences = false, bool $onlyAbsoluteReferences = false)
     {
         // Is it in another worksheet? Will not have to update anything.
         if (strpos($cellReference, '!') !== false) {
@@ -863,11 +867,11 @@ private function updateCellReference($cellReference = 'A1', bool $includeAbsolut
         // Is it a range or a single cell?
         if (!Coordinate::coordinateIsRange($cellReference)) {
             // Single cell
-            return $this->cellReferenceHelper->updateCellReference($cellReference, $includeAbsoluteReferences);
+            return $this->cellReferenceHelper->updateCellReference($cellReference, $includeAbsoluteReferences, $onlyAbsoluteReferences);
         }
 
         // Range
-        return $this->updateCellRange($cellReference, $includeAbsoluteReferences);
+        return $this->updateCellRange($cellReference, $includeAbsoluteReferences, $onlyAbsoluteReferences);
     }
 
     /**
@@ -922,7 +926,7 @@ private function updateNamedRange(DefinedName $definedName, Worksheet $worksheet
              *      them with a #REF!
              */
             if ($asFormula === true) {
-                $formula = $this->updateFormulaReferences($cellAddress, $beforeCellAddress, $numberOfColumns, $numberOfRows, $worksheet->getTitle(), true);
+                $formula = $this->updateFormulaReferences($cellAddress, $beforeCellAddress, $numberOfColumns, $numberOfRows, $worksheet->getTitle(), true, true);
                 $definedName->setValue($formula);
             } else {
                 $definedName->setValue($this->updateCellReference(ltrim($cellAddress, '='), true));
@@ -953,7 +957,7 @@ private function updateNamedFormula(DefinedName $definedName, Worksheet $workshe
      *
      * @return string Updated cell range
      */
-    private function updateCellRange(string $cellRange = 'A1:A1', bool $includeAbsoluteReferences = false): string
+    private function updateCellRange(string $cellRange = 'A1:A1', bool $includeAbsoluteReferences = false, bool $onlyAbsoluteReferences = false): string
     {
         if (!Coordinate::coordinateIsRange($cellRange)) {
             throw new Exception('Only cell ranges may be passed to this method.');
@@ -967,14 +971,14 @@ private function updateCellRange(string $cellRange = 'A1:A1', bool $includeAbsol
             for ($j = 0; $j < $jc; ++$j) {
                 if (ctype_alpha($range[$i][$j])) {
                     $range[$i][$j] = Coordinate::coordinateFromString(
-                        $this->cellReferenceHelper->updateCellReference($range[$i][$j] . '1', $includeAbsoluteReferences)
+                        $this->cellReferenceHelper->updateCellReference($range[$i][$j] . '1', $includeAbsoluteReferences, $onlyAbsoluteReferences)
                     )[0];
                 } elseif (ctype_digit($range[$i][$j])) {
                     $range[$i][$j] = Coordinate::coordinateFromString(
-                        $this->cellReferenceHelper->updateCellReference('A' . $range[$i][$j], $includeAbsoluteReferences)
+                        $this->cellReferenceHelper->updateCellReference('A' . $range[$i][$j], $includeAbsoluteReferences, $onlyAbsoluteReferences)
                     )[1];
                 } else {
-                    $range[$i][$j] = $this->cellReferenceHelper->updateCellReference($range[$i][$j], $includeAbsoluteReferences);
+                    $range[$i][$j] = $this->cellReferenceHelper->updateCellReference($range[$i][$j], $includeAbsoluteReferences, $onlyAbsoluteReferences);
                 }
             }
         }

tests/PhpSpreadsheetTests/Calculation/Functions/LookupRef/IndirectTest.php

@@ -2,6 +2,7 @@
 
 namespace PhpOffice\PhpSpreadsheetTests\Calculation\Functions\LookupRef;
 
+use PhpOffice\PhpSpreadsheet\NamedFormula;
 use PhpOffice\PhpSpreadsheet\NamedRange;
 use PhpOffice\PhpSpreadsheet\Reader\Xlsx as ReaderXlsx;
 
@@ -176,4 +177,33 @@ public static function providerRelative(): array
             'uninitialized cell' => [null, 'RC[+2]'], // Excel result is 0
         ];
     }
+
+    /** @var bool */
+    private static $definedFormulaWorking = false;
+
+    public function testAboveCell(): void
+    {
+        $spreadsheet = $this->getSpreadsheet();
+        $spreadsheet->addNamedFormula(
+            new NamedFormula('SumAbove', $spreadsheet->getActiveSheet(), '=SUM(INDIRECT(ADDRESS(1,COLUMN())):INDIRECT(ADDRESS(ROW()-1,COLUMN())))')
+        );
+        $sheet = $this->getSheet();
+        $sheet->getCell('A1')->setValue(100);
+        $sheet->getCell('A2')->setValue(200);
+        $sheet->getCell('A3')->setValue(300);
+        $sheet->getCell('A4')->setValue(400);
+        $sheet->getCell('A5')->setValue(500);
+        $sheet->getCell('A6')->setValue('=SUM(A$1:INDIRECT(ADDRESS(ROW()-1,COLUMN())))');
+        self::AssertSame(1500, $sheet->getCell('A6')->getCalculatedValue());
+        if (self::$definedFormulaWorking) {
+            $sheet->getCell('B1')->setValue(10);
+            $sheet->getCell('B2')->setValue(20);
+            $sheet->getCell('B3')->setValue(30);
+            $sheet->getCell('B4')->setValue(40);
+            $sheet->getCell('B5')->setValue('=SumAbove');
+            self::AssertSame(100, $sheet->getCell('B5')->getCalculatedValue());
+        } else {
+            self::markTestIncomplete('PhpSpreadsheet does not handle this correctly');
+        }
+    }
 }

tests/PhpSpreadsheetTests/ReferenceHelper3Test.php

@@ -0,0 +1,113 @@
+<?php
+
+namespace PhpOffice\PhpSpreadsheetTests;
+
+use PhpOffice\PhpSpreadsheet\Calculation\Calculation;
+use PhpOffice\PhpSpreadsheet\NamedRange;
+use PhpOffice\PhpSpreadsheet\Spreadsheet;
+use PHPUnit\Framework\TestCase;
+
+class ReferenceHelper3Test extends TestCase
+{
+    public function testIssue3661(): void
+    {
+        $spreadsheet = new Spreadsheet();
+        $sheet = $spreadsheet->getActiveSheet();
+        $sheet->setTitle('Data');
+
+        $spreadsheet->addNamedRange(new NamedRange('FIRST', $sheet, '=$A1'));
+        $spreadsheet->addNamedRange(new NamedRange('SECOND', $sheet, '=$B1'));
+        $spreadsheet->addNamedRange(new NamedRange('THIRD', $sheet, '=$C1'));
+
+        $sheet->fromArray([
+            [1, 2, 3, '=FIRST', '=SECOND', '=THIRD', '=10*$A1'],
+            [4, 5, 6, '=FIRST', '=SECOND', '=THIRD'],
+            [7, 8, 9, '=FIRST', '=SECOND', '=THIRD'],
+        ]);
+
+        $sheet->insertNewRowBefore(1, 4);
+        $sheet->insertNewColumnBefore('A', 1);
+        self::assertSame(1, $sheet->getCell('E5')->getCalculatedValue());
+        self::assertSame(5, $sheet->getCell('F6')->getCalculatedValue());
+        self::assertSame(9, $sheet->getCell('G7')->getCalculatedValue());
+        self::assertSame('=10*$B5', $sheet->getCell('H5')->getValue());
+        self::assertSame(10, $sheet->getCell('H5')->getCalculatedValue());
+        $firstColumn = $spreadsheet->getNamedRange('FIRST');
+        /** @var NamedRange $firstColumn */
+        self::assertSame('=$B1', $firstColumn->getRange());
+        $spreadsheet->disconnectWorksheets();
+    }
+
+    public function testCompletelyRelative(): void
+    {
+        $spreadsheet = new Spreadsheet();
+        $sheet = $spreadsheet->getActiveSheet();
+        $sheet->setTitle('Data');
+
+        $spreadsheet->addNamedRange(new NamedRange('CellAbove', $sheet, '=A1048576'));
+        $spreadsheet->addNamedRange(new NamedRange('CellBelow', $sheet, '=A2'));
+        $spreadsheet->addNamedRange(new NamedRange('CellToLeft', $sheet, '=XFD1'));
+        $spreadsheet->addNamedRange(new NamedRange('CellToRight', $sheet, '=B1'));
+
+        $sheet->fromArray([
+            [null, 'Above', null, null, 'Above', null, null, 'Above', null, null, 'Above', null],
+            ['Left', '=CellAbove', 'Right', 'Left', '=CellBelow', 'Right', 'Left', '=CellToLeft', 'Right', 'Left', '=CellToRight', 'Right'],
+            [null, 'Below', null, null, 'Below', null, null, 'Below', null, null, 'Below', null],
+        ], null, 'A1', true);
+        self::assertSame('Above', $sheet->getCell('B2')->getCalculatedValue());
+        self::assertSame('Below', $sheet->getCell('E2')->getCalculatedValue());
+        self::assertSame('Left', $sheet->getCell('H2')->getCalculatedValue());
+        self::assertSame('Right', $sheet->getCell('K2')->getCalculatedValue());
+
+        Calculation::getInstance($spreadsheet)->flushInstance();
+        self::assertNull($sheet->getCell('L7')->getCalculatedValue(), 'value in L7 after flush is null');
+        // Reset it once more
+        Calculation::getInstance($spreadsheet)->flushInstance();
+        // shift 5 rows down and 1 column to the right
+        $sheet->insertNewRowBefore(1, 5);
+        $sheet->insertNewColumnBefore('A', 1);
+
+        self::assertSame('Above', $sheet->getCell('C7')->getCalculatedValue()); // Above
+        self::assertSame('Below', $sheet->getCell('F7')->getCalculatedValue());
+        self::assertSame('Left', $sheet->getCell('I7')->getCalculatedValue());
+        self::assertSame('Right', $sheet->getCell('L7')->getCalculatedValue());
+
+        $sheet2 = $spreadsheet->createSheet();
+        $sheet2->setCellValue('L6', 'NotThisCell');
+        $sheet2->setCellValue('L7', '=CellAbove');
+        self::assertSame('Above', $sheet2->getCell('L7')->getCalculatedValue(), 'relative value uses cell on worksheet where name is defined');
+        $spreadsheet->disconnectWorksheets();
+    }
+
+    /** @var bool */
+    private static $sumFormulaWorking = false;
+
+    public function testSumAboveCell(): void
+    {
+        $spreadsheet = new Spreadsheet();
+        $sheet = $spreadsheet->getActiveSheet();
+        $spreadsheet->addNamedRange(new NamedRange('AboveCell', $sheet, 'A1048576'));
+        $sheet->setCellValue('C2', 123);
+        $sheet->setCellValue('C3', '=AboveCell');
+        $sheet->fromArray([
+            ['Column 1', 'Column 2'],
+            [2, 1],
+            [4, 3],
+            [6, 5],
+            [8, 7],
+            [10, 9],
+            [12, 11],
+            [14, 13],
+            [16, 15],
+            ['=SUM(A2:AboveCell)', '=SUM(B2:AboveCell)'],
+        ], null, 'A1', true);
+        self::assertSame(123, $sheet->getCell('C3')->getCalculatedValue());
+        if (self::$sumFormulaWorking) {
+            self::assertSame(72, $sheet->getCell('A10')->getCalculatedValue());
+        } else {
+            $spreadsheet->disconnectWorksheets();
+            self::markTestIncomplete('PhpSpreadsheet does not handle this correctly');
+        }
+        $spreadsheet->disconnectWorksheets();
+    }
+}