koboldcpp web server

[Struxxffs]

Hi everyone,

I have been looking into learning how to host local ai.


Currently interested in using koboldcpp

https://github.com/LostRuins/koboldcpp


This might be a stupid question.


Do I need to do anything to stop koboldcpp from exposing its web server outside the home network?


If used on a device that is connected to the internet?

 

[Lord Evermore]

Your normal home router's NAT functionality would prevent any inbound connections. Nobody would even know the web server exists unless you put in a port forward on the router specifically to make the server accessible from the outside. Even if IPv6 is enabled, the router isn't going to forward incoming connections or port scans. (koboldcpp may have a UPNP option to make your router enable port forwarding, but if so it should ask and you can say no, and you should make sure that's disabled on your router in general anyway.)

 

[Struxxffs]

Your normal home router's NAT functionality would prevent any inbound connections. Nobody would even know the web server exists unless you put in a port forward on the router specifically to make the server accessible from the outside. Even if IPv6 is enabled, the router isn't going to forward incoming connections or port scans. (koboldcpp may have a UPNP option to make your router enable port forwarding, but if so it should ask and you can say no, and you should make sure that's disabled on your router in general anyway.)

UPNP appears to be disabled, or not enabled by default on the router. Flint 2 openwrt.
If I avoid using 0.0.0.0, bind it to 127.0.0.1, don't enable port forwarding then it will stay local and not detectable outside of the local machine hosting it?

 

[Lord Evermore]

I suppose, if you don't even want anything on the local LAN to access it.

 

[steelghost]

This is what firewalls (should) do - mean that nothing on the internet can just randomly connect to a port on a device on your LAN, without that device having first made a connection to the computer on the internet. So you can visit a web page and get the payload from the webserver for your browser to render, but not the other way around unless you deliberaltely set it up that way.

As you say, if you bind to localhost then nothing on the LAN would be able to access it either. Do you have IPv6 on your LAN? If so make sure you're binding to localhost (::1) for that as well - otherwise a LAN device could access it via IPv6.

 

[Struxxffs]

I suppose, if you don't even want anything on the local LAN to access it.

Currently, I do not. Still learning and trying to make sure to follow proper security and keep web server apps locked down. It would be easier if the machine was completely offline, but that is not really convenient.

This is what firewalls (should) do - mean that nothing on the internet can just randomly connect to a port on a device on your LAN, without that device having first made a connection to the computer on the internet. So you can visit a web page and get the payload from the webserver for your browser to render, but not the other way around unless you deliberaltely set it up that way.

As you say, if you bind to localhost then nothing on the LAN would be able to access it either. Do you have IPv6 on your LAN? If so make sure you're binding to localhost (::1) for that as well - otherwise a LAN device could access it via IPv6.

Thank you.

So as long as I don’t set up port forwarding or open up firewall rules, nothing from the internet can connect to KoboldCpp, even though the machine has internet access — since it’s not making an outbound connection first.

I do have IPv6 on the lan.


This github post on the project is what caused some concern:
https://github.com/LostRuins/koboldcpp/issues/539

Spoiler

Koboldcpp sets the CORS header access-control-allow-origin: *, which allows any JavaScript run on any of the user's web browsers to connect to the localhost:5001 endpoint.


E.g. if you browse to some random web site in Chrome, the following can happen:

1. The web site runs (untrusted) JavaScript code in your browser, as most web sites these days do
2. The JavaScript code attempts to connect to localhost:5001
3. Your browser runs locally and has access to localhost:5001 regardless of any firewalls or the host setting
4. The JavaScript runs in your browser, so it runs locally too, and has access to localhost:5001 if the browser allows it
5. The browser tries to connect localhost:5001 and asks the endpoint if it's OK to connect. It does this by checking the CORS header.
6. Koboldcpp responds with the CORS header access-control-allow-origin: *, which means "yes, anyone can connect!"
7. The web browser allows connection from the (untrusted) JavaScript code to localhost:5001
8. The JavaScript code can also communicate with the remote site it was loaded from. This means that the owner of the remote site can open a connections to your localhost:5001 as long as you keep the web site open in your browser. The JavaScript code works as a proxy in this scenario.

You can e.g. try visiting this web site with koboldcpp running, and it will tell you that it can see localhost:5001.


Note also that the current default settings are to listen on all interfaces. Any computer on the local network that can connect to the port 5001 on your machine is allowed access by CORS. This means that any web sites anyone on those computers visits can connect too — their browser is not going to prevent the connection.

The most important part of it was this:

Spoiler

You can e.g. try visiting this web site with koboldcpp running, and it will tell you that it can see localhost:5001.


Note also that the current default settings are to listen on all interfaces. Any computer on the local network that can connect to the port 5001 on your machine is allowed access by CORS. This means that any web sites anyone on those computers visits can connect too — their browser is not going to prevent the connection.

 

[Lord Evermore]

You can e.g. try visiting this web site with koboldcpp running, and it will tell you that it can see localhost:5001.

They're running JavaScript in the browser on the computer that is running koboldcpp, and the "app" inside the browser tries to connect to localhost:5001. The hostname "localhost" by default points to 127.0.0.1 on all network devices. The "access-control-allow-origin" header tells koboldcpp to allow the connection.

By using Javascript, it's possible to basically use the browser as a proxy for the owner of the remote website. The remote site doesn't have direct access to your koboldcpp instance. The remote site talks to the JavaScript app in the browser, and the browser relays whatever requests are made to the koboldcpp instance, and then relays the responses back to the remote site. But the remote site can't do anything unless you've opened that page that loads that JavaScript code, and keep it open.

In this instance, they're demonstrating it working specifically to connect to koboldcpp running on the same machine as the web browser, specifically to the 127.0.0.1 IP. But this isn't something that would be restricted to koboldcpp, or only on the local host or that port. It could be done to access any listening ports on any machine in your local network. Heck, it could be used to make your machine act as a proxy to connect to other Internet servers, allowing a bad actor to do things and make it look like it was coming from your computer/IP, and then they could wipe all trace of their activity from your system since it was just a JavaScript and they could prevent anything being cached or clear the history.

The access-control-allow-origin header can be modified to prevent access from any but sources that you want to allow, in any web server. I suppose the way that koboldcpp has it configured could be considered "insecure" but really only if you weren't taking OTHER precautions. If it's only listening on 127.0.0.1 then only requests from the local machine would be allowed, and only a very specific method like this JavaScript page could be used which would require you to be browsing to a site with the code from that same machine to abuse it.

 

[Struxxffs]

They're running JavaScript in the browser on the computer that is running koboldcpp, and the "app" inside the browser tries to connect to localhost:5001. The hostname "localhost" by default points to 127.0.0.1 on all network devices. The "access-control-allow-origin" header tells koboldcpp to allow the connection.

By using Javascript, it's possible to basically use the browser as a proxy for the owner of the remote website. The remote site doesn't have direct access to your koboldcpp instance. The remote site talks to the JavaScript app in the browser, and the browser relays whatever requests are made to the koboldcpp instance, and then relays the responses back to the remote site. But the remote site can't do anything unless you've opened that page that loads that JavaScript code, and keep it open.

In this instance, they're demonstrating it working specifically to connect to koboldcpp running on the same machine as the web browser, specifically to the 127.0.0.1 IP. But this isn't something that would be restricted to koboldcpp, or only on the local host or that port. It could be done to access any listening ports on any machine in your local network. Heck, it could be used to make your machine act as a proxy to connect to other Internet servers, allowing a bad actor to do things and make it look like it was coming from your computer/IP, and then they could wipe all trace of their activity from your system since it was just a JavaScript and they could prevent anything being cached or clear the history.

The access-control-allow-origin header can be modified to prevent access from any but sources that you want to allow, in any web server. I suppose the way that koboldcpp has it configured could be considered "insecure" but really only if you weren't taking OTHER precautions. If it's only listening on 127.0.0.1 then only requests from the local machine would be allowed, and only a very specific method like this JavaScript page could be used which would require you to be browsing to a site with the code from that same machine to abuse it.

Thank you for the explanation. It sounds like the real risk isn’t from the remote website itself directly connecting to KoboldCPP, but from the JavaScript it runs in my own browser, acting as a kind of bridge while the page is open.

As long as its running 127.0.0.1 and only browsing secure websites the risk is pretty minimal.

 

[Lord Evermore]

As long as its running 127.0.0.1 and only browsing secure websites the risk is pretty minimal.

Essentially, although the problem is ensuring the site you're browsing is "secure", given how very creative bad actors are at finding ways to insert their code to load in your browser. Ads that deliver code is an old one, but at a deeper level there's always the risk of a site being recently compromised and not having reacted yet, and worse, one that thinks it's secure but has actually been compromised for a long period. Or the site itself is secure but some component they get from elsewhere isn't (supply chain attack). All the way to compromised packages loaded by the OS.

 

[Struxxffs]

Essentially, although the problem is ensuring the site you're browsing is "secure", given how very creative bad actors are at finding ways to insert their code to load in your browser. Ads that deliver code is an old one, but at a deeper level there's always the risk of a site being recently compromised and not having reacted yet, and worse, one that thinks it's secure but has actually been compromised for a long period. Or the site itself is secure but some component they get from elsewhere isn't (supply chain attack). All the way to compromised packages loaded by the OS.

You bring up good points. It sounds like the secure way then is to not browse other sites while using koboldcpp.

 

[Lord Evermore]

You bring up good points. It sounds like the secure way then is to not browse other sites while using koboldcpp.

That's a bit extreme. There's risk involved just from turning on a computer, with any OS these days, if you are going to have a network connection. Doesn't even have to have access to the Internet itself. Just being on a network where OTHER devices exist is a risk because one of those could be infected or otherwise controlled by someone malicious, and stupid stuff like SMB flaws allow infections without even DOING anything. Heck, plug in a USB drive that's never been connected to another PC and you might get infected with something that was on the drive from the factory due to a supply chain attack.

It's always a trade-off to get the things you want/need out of a machine, balancing convenience and functionality with security. Completely avoiding doing anything else on the koboldcpp machine would make it much more inconvenient to use koboldcpp itself (I don't really know WHAT you can do with it anyway or how you do so), maybe even impossible, and just following best practices of not going to questionable sites that might be willing to be involved with other risky services or even be run by bad actors directly (porn, content pirating, gambling, etc.) and using ad-blockers and script blocking is generally enough to ensure you're not going to have a problem. And a site trying to run JS to access your koboldcpp server is VERY niche so the odds are much lower, although if you're searching for information about that software you might be more likely to run across it. Just stick to well-known and trusted sites and you'll probably be okay, but you might be the one in 10 million to get infected. If koboldcpp is going to be processing your company financial data or all the naughty pictures and videos you made with all your exes then you'd want to limit what you do with that machine, but otherwise treat it like any other computer where you'd do things like entering passwords on websites, accessing your bank account or healthcare info or browsing content you wouldn't want your pastor to know about.

 

[Struxxffs]

That's a bit extreme. There's risk involved just from turning on a computer, with any OS these days, if you are going to have a network connection. Doesn't even have to have access to the Internet itself. Just being on a network where OTHER devices exist is a risk because one of those could be infected or otherwise controlled by someone malicious, and stupid stuff like SMB flaws allow infections without even DOING anything. Heck, plug in a USB drive that's never been connected to another PC and you might get infected with something that was on the drive from the factory due to a supply chain attack.

It's always a trade-off to get the things you want/need out of a machine, balancing convenience and functionality with security. Completely avoiding doing anything else on the koboldcpp machine would make it much more inconvenient to use koboldcpp itself (I don't really know WHAT you can do with it anyway or how you do so), maybe even impossible, and just following best practices of not going to questionable sites that might be willing to be involved with other risky services or even be run by bad actors directly (porn, content pirating, gambling, etc.) and using ad-blockers and script blocking is generally enough to ensure you're not going to have a problem. And a site trying to run JS to access your koboldcpp server is VERY niche so the odds are much lower, although if you're searching for information about that software you might be more likely to run across it. Just stick to well-known and trusted sites and you'll probably be okay, but you might be the one in 10 million to get infected. If koboldcpp is going to be processing your company financial data or all the naughty pictures and videos you made with all your exes then you'd want to limit what you do with that machine, but otherwise treat it like any other computer where you'd do things like entering passwords on websites, accessing your bank account or healthcare info or browsing content you wouldn't want your pastor to know about.

Knowing there is a low chance of infection is a good thing to hear. Basically use common sense for browsing, follow security practices that have been suggested, and don't over think it.