ESP8266WebServerSecure: streamFile not correctly implemented

[erics465]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP8266, NodeMCU 1.0 (ESP-12E Module)
• Core Version: 2.4.1 and current commit (#2013af1), both tested
• Development Env: Arduino IDE
• Operating System: Windows

Settings in IDE

• Module: NodeMCU 1.0 (ESP-12E Module)
• Flash Mode: qio
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

ESP8266WebServerSecure uses method streamFile of ESP8266WebServer

ESP8266WebServer and ESP8266WebServerSecure provide both the method streamFile which can stream a file to a client that connected to the webserver. However, this only works with unencrypted webservers based on ESP8266WebServer. Class ESP8266WebServerSecure doesn't overwrite the original method that is inherited from ESP8266WebServer. This obviously doesn't work with encrypted sockets, as it tries to send the date from the file a) to the standard socket from ESP8266WebServer and b) it doesn't encrypt the data.

Changes need to be made in https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WebServer/src/ESP8266WebServerSecure.h and https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WebServer/src/ESP8266WebServerSecure.cpp which need to a) overwrite the standard streamFile method and b) provide a way to pass streams to the ssl library. Unfortunately, I'm not that experienced with C/C++ and cannot implement the changes on my own. That's the reason why I'm asking here to fix this issue.

The example creates a demo file "test.html" that is sent to the client and includes a 512 bit demo ssl certificate, which may get rejected when testing with recent browsers such as Google Chrome. Use command line tools like wget with the --no-check-certificate option to connect to the server.

A demo output from wget below shows that no data is send, which lets the request fail.

MCVE Sketch

#include <ESP8266WiFi.h>
#include <FS.h>
#include <ESP8266WebServerSecure.h>

//wifi credentials
const char* ssid = "CHANGE ME";
const char* password = "CHANGE ME";

// The certificate is stored in PMEM
static const uint8_t x509[] PROGMEM = {
  0x30, 0x82, 0x01, 0x51, 0x30, 0x81, 0xfc, 0x02, 0x09, 0x00, 0xa6, 0x59,
  0xaa, 0xc1, 0x62, 0x1a, 0xd9, 0xbc, 0x30, 0x0d, 0x06, 0x09, 0x2a, 0x86,
  0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x30, 0x30, 0x31,
  0x17, 0x30, 0x15, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x0c, 0x0e, 0x79, 0x6f,
  0x75, 0x72, 0x2d, 0x6e, 0x61, 0x6d, 0x65, 0x2d, 0x68, 0x65, 0x72, 0x65,
  0x31, 0x15, 0x30, 0x13, 0x06, 0x03, 0x55, 0x04, 0x03, 0x0c, 0x0c, 0x65,
  0x73, 0x70, 0x38, 0x32, 0x36, 0x36, 0x2d, 0x64, 0x65, 0x6d, 0x6f, 0x30,
  0x1e, 0x17, 0x0d, 0x31, 0x38, 0x30, 0x33, 0x31, 0x39, 0x31, 0x36, 0x35,
  0x33, 0x30, 0x30, 0x5a, 0x17, 0x0d, 0x33, 0x31, 0x31, 0x31, 0x32, 0x36,
  0x31, 0x36, 0x35, 0x33, 0x30, 0x30, 0x5a, 0x30, 0x30, 0x31, 0x17, 0x30,
  0x15, 0x06, 0x03, 0x55, 0x04, 0x0a, 0x0c, 0x0e, 0x79, 0x6f, 0x75, 0x72,
  0x2d, 0x6e, 0x61, 0x6d, 0x65, 0x2d, 0x68, 0x65, 0x72, 0x65, 0x31, 0x15,
  0x30, 0x13, 0x06, 0x03, 0x55, 0x04, 0x03, 0x0c, 0x0c, 0x65, 0x73, 0x70,
  0x38, 0x32, 0x36, 0x36, 0x2d, 0x64, 0x65, 0x6d, 0x6f, 0x30, 0x5c, 0x30,
  0x0d, 0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01,
  0x05, 0x00, 0x03, 0x4b, 0x00, 0x30, 0x48, 0x02, 0x41, 0x00, 0xc5, 0x5f,
  0x7a, 0xa4, 0xa5, 0x94, 0x45, 0xde, 0x1b, 0x7e, 0x28, 0x07, 0x4f, 0x76,
  0x60, 0xb5, 0xe5, 0xa8, 0xba, 0x98, 0xb7, 0xb8, 0x41, 0x69, 0x44, 0x14,
  0xdf, 0x5b, 0x92, 0xdd, 0x84, 0xac, 0x3d, 0x20, 0x82, 0x51, 0x19, 0x9a,
  0x68, 0xed, 0xca, 0xee, 0x45, 0x95, 0xe5, 0x15, 0xbb, 0x8f, 0xc2, 0xa1,
  0x02, 0xb9, 0x85, 0xb0, 0xfe, 0xfd, 0x9c, 0x55, 0x88, 0x7d, 0x02, 0x50,
  0x8d, 0xdf, 0x02, 0x03, 0x01, 0x00, 0x01, 0x30, 0x0d, 0x06, 0x09, 0x2a,
  0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b, 0x05, 0x00, 0x03, 0x41,
  0x00, 0x03, 0x04, 0x71, 0x08, 0x16, 0x9e, 0x6e, 0x84, 0x9f, 0x1f, 0xed,
  0x8d, 0xa2, 0x94, 0xae, 0x11, 0xa6, 0x64, 0x9d, 0x4c, 0xb1, 0x8a, 0xc5,
  0x17, 0x9c, 0xa9, 0xae, 0x30, 0xa3, 0xa1, 0xb8, 0xa6, 0x49, 0xda, 0xab,
  0xa3, 0x11, 0x0b, 0xbd, 0x2d, 0xfb, 0xd9, 0x34, 0x75, 0x50, 0xf8, 0xa9,
  0x53, 0x9f, 0x12, 0x3d, 0xf7, 0x90, 0x9d, 0xea, 0x7b, 0x96, 0x16, 0x59,
  0x5b, 0xc5, 0x6e, 0xdf, 0x19
};

// And so is the key.  These could also be in DRAM
static const uint8_t rsakey[] PROGMEM = {
  0x30, 0x82, 0x01, 0x3b, 0x02, 0x01, 0x00, 0x02, 0x41, 0x00, 0xc5, 0x5f,
  0x7a, 0xa4, 0xa5, 0x94, 0x45, 0xde, 0x1b, 0x7e, 0x28, 0x07, 0x4f, 0x76,
  0x60, 0xb5, 0xe5, 0xa8, 0xba, 0x98, 0xb7, 0xb8, 0x41, 0x69, 0x44, 0x14,
  0xdf, 0x5b, 0x92, 0xdd, 0x84, 0xac, 0x3d, 0x20, 0x82, 0x51, 0x19, 0x9a,
  0x68, 0xed, 0xca, 0xee, 0x45, 0x95, 0xe5, 0x15, 0xbb, 0x8f, 0xc2, 0xa1,
  0x02, 0xb9, 0x85, 0xb0, 0xfe, 0xfd, 0x9c, 0x55, 0x88, 0x7d, 0x02, 0x50,
  0x8d, 0xdf, 0x02, 0x03, 0x01, 0x00, 0x01, 0x02, 0x40, 0x5c, 0x9d, 0x10,
  0xcf, 0x71, 0x62, 0xc1, 0xe6, 0x16, 0xc0, 0x7b, 0xc7, 0xf2, 0x61, 0x79,
  0xbf, 0xe0, 0xa3, 0xeb, 0xea, 0xfd, 0x4a, 0x58, 0x67, 0x2c, 0xab, 0x1e,
  0xa3, 0xb9, 0xa6, 0x78, 0x3a, 0x06, 0x61, 0xf8, 0xef, 0x2f, 0x97, 0xd5,
  0x3e, 0x24, 0x86, 0x55, 0xcb, 0x5e, 0x21, 0x8d, 0xa3, 0xd6, 0x87, 0xfd,
  0x53, 0xa7, 0xd3, 0xca, 0xe1, 0xba, 0x9d, 0x8c, 0x30, 0x00, 0xf7, 0x25,
  0x01, 0x02, 0x21, 0x00, 0xf2, 0xb0, 0xc0, 0x7a, 0xbe, 0xdc, 0x8b, 0xf0,
  0x2b, 0x7c, 0xfc, 0x1d, 0x1c, 0xd5, 0x9e, 0x3b, 0x0c, 0xda, 0xfd, 0x53,
  0x31, 0x73, 0x89, 0x4f, 0x33, 0x85, 0xc1, 0x52, 0x2b, 0xea, 0x7f, 0xe3,
  0x02, 0x21, 0x00, 0xd0, 0x32, 0x7e, 0x4c, 0x4b, 0x3b, 0x8f, 0x05, 0xaa,
  0x97, 0xf5, 0xf1, 0xd2, 0xcc, 0x67, 0x57, 0x9b, 0x09, 0x40, 0x93, 0x0a,
  0x1f, 0x33, 0xb4, 0x39, 0x66, 0x7f, 0x65, 0x55, 0x68, 0x22, 0xd5, 0x02,
  0x21, 0x00, 0x8e, 0x17, 0x84, 0xaa, 0x99, 0x43, 0x01, 0xbf, 0xdd, 0x86,
  0x71, 0x0a, 0x0e, 0x8e, 0xd7, 0xf4, 0xd4, 0xe3, 0x06, 0xbd, 0x05, 0xd0,
  0x12, 0x8d, 0xc7, 0xa9, 0xc1, 0x75, 0x7d, 0xf6, 0xef, 0x67, 0x02, 0x21,
  0x00, 0xc7, 0x04, 0xfd, 0xa6, 0x80, 0xe4, 0x56, 0x3b, 0xdc, 0x6f, 0x97,
  0x33, 0xab, 0x86, 0xa9, 0xe3, 0x1c, 0xd9, 0x23, 0x59, 0x6b, 0xfb, 0x97,
  0xb9, 0x58, 0x85, 0x9a, 0x92, 0x8b, 0xaa, 0x18, 0x61, 0x02, 0x20, 0x79,
  0xfc, 0x79, 0x86, 0xd1, 0x2f, 0x3a, 0x18, 0x56, 0xb7, 0x16, 0x8a, 0x57,
  0x2f, 0x59, 0x9a, 0x3e, 0x5f, 0x6c, 0x1c, 0x0d, 0x14, 0x28, 0xf7, 0xa6,
  0x84, 0xea, 0x18, 0xee, 0x0d, 0x64, 0x63
};

ESP8266WebServerSecure server(443);

void setup() {
  Serial.begin(115200);
  Serial.println();

  if (!SPIFFS.begin()) {
    Serial.println("Failed to mount file system");
    return;
  }

  //create demo file
  File f = SPIFFS.open("/test.html", "w");
  if (!f) {
    Serial.println("file creation failed");
  }
  f.println("<h1>Hello!</h1>");
  f.close();
  
  WiFi.mode(WIFI_STA);
  int status = WL_IDLE_STATUS;
  while (status != WL_CONNECTED) {
    //connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, password);
    delay(10000);
  }
  Serial.print("Connected!");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  server.setServerKeyAndCert_P(rsakey, sizeof(rsakey), x509, sizeof(x509));
  server.on("/", HTTP_GET, []() {handleFileRead("/test.html");});
  server.onNotFound([]() {handleFileRead("/test.html");});
  server.begin();
}

void loop() {
  server.handleClient();
}

bool handleFileRead(String path) {
  if (SPIFFS.exists(path)) {
    File file = SPIFFS.open(path, "r");
    server.streamFile(file, "text/html");
    file.close();
    return true;
  }
  return false;
}

wget Output

$ wget https://192.168.2.108 --no-check-certificate
https://192.168.2.108/
Connecting to 192.168.2.108:443... connected.
WARNING: cannot verify 192.168.2.108's certificate, issued by ‘CN=esp8266-demo,O=your-name-here’:
  Self-signed certificate encountered.
    WARNING: certificate common name ‘esp8266-demo’ doesn't match requested host name ‘192.168.2.108’.
HTTP request sent, awaiting response... 200 OK
Length: 17 [text/html]
Saving to: ‘index.html.1’

index.html.1                    0%[                                                  ]       0  --.-KB/s    in 1.9s

(0.00 B/s) - Connection closed at byte 0. Retrying.

[earlephilhower]

Looks to be a very simple fix, just need to add the template and make is use _currentClientSecure instead of _currentClient to send the data. I'll throw something together tonight.

[erics465]

Are you sure that this is enough? WifiClientSecure seems to be missing a client.write method that can handle streams, which needs to be implemented. Or am I missing something there?

[earlephilhower]

You might be right, if the template doesn't compile when calling _currentClientSecure.write(stream) that'll need to be added to WiFiClientSecure as well. Nothing conceptually hard, I think.

I'll definitely be sure to test your MCVE sketch!

[earlephilhower]

You were right, the write(Stream&) method wasn't there. It is now, in the PR. I tested your MCVE and it works fine, but if you could test your real app with it that would be a much better test.

[erics465]

Thank you for this wonderful fast fix. My real app doesn't differ that much from the example, but both applications work fine with your pull request. For me, this problem is now fixed. Thanks for your time and keep on the good work. ;)

Should I now close this issue or wait until it is merged?