Photo hosting on MyE28

MyE28.com Forum system comments and questions. Please post registration, login, or general forum usage problems here.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Photo hosting on MyE28

Post by tig »

UPDATE: Jan 26, 2025
  • Added User Control Panel Photo Gallery
  • Added Admin view of uploaded photos
Local Storage of Externally Linked Photos
As part of the infrastructure upgrade, every external link via an img tag to a photo posted prior to December 27, 2024 is now automatically be redirected to a copy of the image hosted by mye28.com.

That's 149,696 photos using 57GB of storage that will never be lost if services like Photofuckit go tits up, or members stop paying their dues to said services.

The photos are stored in an Azure Blob Storage Container (the MS equivalent of an Amazon S3 Bucket for those who follow these things). That Blob Container is mounted on the mye28.com server as a file system, enabling file system access.

Prior to early Jan, when you viewed a post, such images were pulled from a URL like this:

Code: Select all

https://www.mye28.com/images/ext/0b3dddde2eef0a2b81e4112f68bc2280.jpg
IOW, they were served up by the mye28.com webserver.

As early Jan, they are now pulled directly from the Blob Bucket, with URLs like this:

Code: Select all

https://mye28.z13.web.core.windows.net/external/0b3dddde2eef0a2b81e4112f68bc2280.jpg
IOW, the Internet address of the Blob Container is https://mye28.z13.web.core.windows.net.

The great things about this scheme are:

- Storage costs are extremely low (roughly $0.020 per GB/month).
- There's no load on the mye28.com virtual machine when photos are accessed.
- It's trivial to setup inexpensive CDN support

Wiki Photos and Files
Separately, the wiki at https://wiki.mye28.com has its own image/file store. While the Wiki is hosted on the same virtual machine as the forums, the storage is separate. I've enabled uploads of images (and some other file types like PDF and Excel) for all users. Fles uploaded via the Wiki uploader are stored on the VM hard disk for now.

However, I've enabled the Wiki to access the same Blob Container and am already using it for large files (all the ETMs at https://wiki.mye28.com/index.php/Electrical). A future work item is to make the standard WIki uploader put images in the Blob Container not local storage.

Please DO NOT use the WIki uploader as a way of hosting photos you put into mye28 posts at this time!

The New Uploader
As of January 24, 2025 when members access the Post Editor they will see a new "Uploader" tab below the Post Editor, like this:

Image

This is the current interface for the Uploader.

To drag & drop to upload:

Simply drag photos onto the editor and they will be uploaded and the appropriate bbcodes will be added to the post.

To manually select files to upload
  1. Click on the Uploader tab.
  2. Click on "Choose Files"
  3. Pick any number of files from the PC or phone you are using (or even take a new photo with your phone).
  4. The photos you selected will be uploaded and displayed, like this:
ImageFrom here you have several choices on how to put the links into your post:

Copy/Paste BBCodes
  • Click on "Click here to copy the BBcodes for all uploaded images to the clipboard" to copy all the BBCodes for the files you uploaded.
  • Click on the Copy Icon next to the BBCodes for an individual photo. This will copy that BBCode to the clipboard.
To paste BBCodes into the editor, you must either be using the non-WYSIWYG editor or you must put the WYSISWIG editor in Source mode. This toolbar button puts the WYSIWYG editor into source mode:
Image
Copy/Paste Image URL
  1. Click on the Copy Icon next to the image URL for an individual icon. This will copy that URLto the clipboard.
  2. If using the WYSISWIG editor, click on the Image button and paste the URL into the Link box.
Image
Image
The advantage of using the BBcodes is if a user clicks on the image in the link, they'll be taken to the 4K resolution version.

View Your Photos in the User Control Panel UCP
Head to your UCP and you'll see a "Uploader" tab. Click on it and you'll get a photo gallery of all the photos you've uploaded.


  Image

  • Click on any thumbnail to got to the full-resolution original image.
  • Click on the "Copy BBcode" and the BBcodes will be copied to the clipboard for easy pasting into a post.
Cool Features
Here are some of the cool features of this Uploader:
  • Photos are stored per-user. This enables a few things including: Being able to identify who uploaded up as a way of reducing abuse and the ability to enable per-user photo galleries in the future.
  • HEIC files are supported. iPhone users, know that Apple stores photos on the phone in a proprietary image format called HEIC. It's been the bane of users' existence trying to convert HEIC files to JPG or PNG so they work well with others. The MyE28 uploader fully supports automatic conversion of HEIC files to JPGs. Magic.
  • Uploaded image size is limited to 4K resolution (3840x2160), but larger images will automagically downsized to that resolution. 
  • Images displayed (by default) in posts are sized at 1280x1024 for efficient viewing, but the original (4K max) image is always available.
  • GIFs are fully supported.
  • The photo storage capacity is basically infinite and very cheap. Upload away!
Next Steps
What's deployed now is just the beginning of my ambition for the Uploader. It has some limitations I plan on fixing over time:
  • The ugly https://mye28.z13.web.core.windows.net/uploads URLs are ugly. It should be possible for https://photos.mye28.com/uploads to work.
  • Copy/Paste is cumbersome. I want to implement two features to make this easier:
    • Make the Image toolbar button support uploading files from within it (and showing previously uploaded files)..
  • Regular task that will run ensuring any photos that are not linked from a post get deleted. 
  • Performance is just OK. The time it takes for a photo upload is about the same as most other sites (actually faster than Flickr), but it could be faster. I haz ideas.
Give Feedback
Please don't be shy about using this thread to ask questions and give feedback on the experience. 
1st 5er
Posts: 21870
Joined: Jun 13, 2008 12:15 AM
Location: Cypress
Contact:

Re: Photo hosting on MyE28

Post by 1st 5er »

I love you tig. :pimp:
vinceg101
Posts: 4899
Joined: Jun 20, 2007 2:40 AM
Location: Los Angeles, CA

Re: Photo hosting on MyE28

Post by vinceg101 »

Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?

As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).
I have my own Nextcloud server that I just figured out how to host photos here and could migrate all the photos I have here to it so I could effectively drop PB but that is going to take a really long time to do. I am wondering how this is going to effect that.

Otherwise:
Image
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

vinceg101 wrote: Dec 31, 2024 1:30 PM Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?
Not yet.
vinceg101 wrote: Dec 31, 2024 1:30 PM As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).
If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

tig wrote: Dec 31, 2024 1:54 PM
vinceg101 wrote: Dec 31, 2024 1:30 PM As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).
If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.
Vince, I just went through your thread

viewtopic.php?t=139465

and it appears all photos are working. Some are still coming from Photofuckit.

I think I need to run the script again to pick up more PB files.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

tig wrote: Dec 31, 2024 2:14 PM
tig wrote: Dec 31, 2024 1:54 PM

If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.
Vince, I just went through your thread

viewtopic.php?t=139465

and it appears all photos are working. Some are still coming from Photofuckit.

I think I need to run the script again to pick up more PB files.
Found the problem. The script didn't deal with image URLs with query strings. These worked:

Code: Select all

[IMG]https://hosting.photobucket.com/albums/rr353/vinceg101/M535%20Front%20End%20Overhaul/.highres/FEO_Ex7.jpg[/IMG]
[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0511.JPG[/IMG]
These didnt:

Code: Select all

[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0511.JPG?width=1920&height=1080&fit=bounds[/IMG]
[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0012(1).JPG?width=1920&height=1080&fit=bounds[/IMG]
I'm now re-running the scripts. It may take a day or two...
vinceg101
Posts: 4899
Joined: Jun 20, 2007 2:40 AM
Location: Los Angeles, CA

Re: Photo hosting on MyE28

Post by vinceg101 »

I just went through that whole thread and didn't see any missing links either (all of those are on PB). So that one isn't a problem.
I will check my other threads (I could have sworn there were missing photos somewhere).
It may take a bit (guess I know what I'm doing while watching football tomorrow...)
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

tig wrote: Dec 31, 2024 1:54 PM
vinceg101 wrote: Dec 31, 2024 1:30 PM Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?
Not yet. 
 
Vince, if I can get the new post editor to work as advertised, the process will be as easy as dragging one or more photos from your Finder/Explorer into the editor, exactly as you’d do with a word processor. 
milarsky
Posts: 525
Joined: Jan 26, 2013 11:19 AM
Location: NYC

Re: Photo hosting on MyE28

Post by milarsky »

My goodness.  Thank you!!!!!  I'm so excited for the best BMW board by far.

jeff
e28Sean
Posts: 993
Joined: Nov 23, 2007 2:07 PM
Location: Greenville, SC
Contact:

Re: Photo hosting on MyE28

Post by e28Sean »

Thank you again for all the hard work you've put in.   I'm going to put out one word of caution, though; Keep in mind the liability involved in hosting images if someone uploads something illegal.  You don't want to find the site in possession of that sort of thing if the law comes knocking.

I only bring it up because I've had an issue on another site I'm involved with, wherein we had a member uploading things that were illegal in most universes.  We ended up killing all image hosting and telling our users to use something like Imgur.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

e28Sean wrote: Jan 02, 2025 9:54 AM Thank you again for all the hard work you've put in. I'm going to put out one word of caution, though; Keep in mind the liability involved in hosting images if someone uploads something illegal. You don't want to find the site in possession of that sort of thing if the law comes knocking.

I only bring it up because I've had an issue on another site I'm involved with, wherein we had a member uploading things that were illegal in most universes. We ended up killing all image hosting and telling our users to use something like Imgur.
Understood.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

vinceg101 wrote: Dec 31, 2024 7:43 PM I just went through that whole thread and didn't see any missing links either (all of those are on PB). So that one isn't a problem.
I will check my other threads (I could have sworn there were missing photos somewhere).
It may take a bit (guess I know what I'm doing while watching football tomorrow...)
The fixed external images script just completed. Vince, your thread we were using as test above is now fixed.

All images come from the image-redirect store (http://mye28.z1...) now:

Re: 1985 Arktisblau M535i Renovation - Progression - MyE28.com
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

Previously there were 150571 images in the image-redirect (external) store. Now there are 156581.

Storage used is about 81GB.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

If you think an externally linked photo should be working but isn't I really want to know about it!

For example, if you see this: 

Image

You can do the following to help diagnose:

Right click on "Image" and select you browser's equivalent of "Inspect":
Image

You'll see something like this:
Image

The browser is showing you the underling HTML. In this case an <img> tag linking to an external photo at "http://memimage.cardomain.net/member_im ... 5_full.jpg".

Right click on that URL and select "Open in new Tab". In this example, if you do that you'll get an error that the image doesn't exist. This is because cardomain.net died a horrible death years ago. This image is GONE. So there's no fixing it.

If you "Open in new Tab" and the expected image shows up, please post to this thread:

- A link to the Post in question.
- The URL of the missing photo.

Thanks!

 
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

This is a test. This image was uploaded using the built-in PhpBB attachments feature. Which I temporarily enabled.
mye28.png
 
Justin_FL
MyE28 IT Guru
MyE28 IT Guru
Posts: 2837
Joined: Feb 12, 2006 12:00 PM
Location: Palm Beach
Contact:

Re: Photo hosting on MyE28

Post by Justin_FL »

Looks like some pictures from my 518i thread are not redirecting to the MyE28 image storage. Seems to only affect the pictures on Google photos, but only a couple as most pulling from our server.

Thread: viewtopic.php?t=115493

Example image:
Image
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

Yup, a bunch of photos from googleusercontent.com didn't get downloaded by the script.

Fixing now...

I guess I need do figure out how to test this more deeply. 
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

Ok googleusercontent.com should all be downloaded now, if they are good URLs.

 
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

The external image cache now has 212577 photos.
DavidE9
Posts: 263
Joined: Apr 22, 2012 1:29 AM
Location: 34.138016, -117.214714

Re: Photo hosting on MyE28

Post by DavidE9 »

Sorry if I missed it, but how do we post photos going forward ? Can they be directly uploaded to the server, or do we need to use a 3rd party ?

Thanks again for the work here.
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

DavidE9 wrote: Jan 05, 2025 7:52 PM Sorry if I missed it, but how do we post photos going forward ? Can they be directly uploaded to the server, or do we need to use a 3rd party ?

Thanks again for the work here.
If you read the first post of this thread you'd see that you still need to use a 3rd party. Built-in hosting is a work-in-progress.
ahab
Posts: 6201
Joined: Jun 11, 2006 9:12 AM
Location: Chalfont, PA

Re: Photo hosting on MyE28

Post by ahab »

So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?
Justin_FL
MyE28 IT Guru
MyE28 IT Guru
Posts: 2837
Joined: Feb 12, 2006 12:00 PM
Location: Palm Beach
Contact:

Re: Photo hosting on MyE28

Post by Justin_FL »

ahab wrote: Jan 06, 2025 12:52 PM So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?
That matches my understanding of the current setup. The built-in phpBB attachment function leaves a lot to be desired and at least with the old server it was not practical to host gigs of pictures, so just flipping the switch wasn't as simple as it sounds. I'm glad Tig has the time, skills and enthusiasm to make something better than the defaults with the power of the new setup!
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

Justin_FL wrote: Jan 06, 2025 1:38 PM
ahab wrote: Jan 06, 2025 12:52 PM So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?
That matches my understanding of the current setup. The built-in phpBB attachment function leaves a lot to be desired and at least with the old server it was not practical to host gigs of pictures, so just flipping the switch wasn't as simple as it sounds. I'm glad Tig has the time, skills and enthusiasm to make something better than the defaults with the power of the new setup!
This is not quite right. There are two scenarios:

Legacy External Photo Links
I've added an extension to the forum called image-redirect that includes scripts that, when run, go through every post on the board, identifying img tags that link to photos. For each img tag, the original external photo is downloaded to mye28's storage. Then, anytime one of those posts are accessed the etension changes the HTML served to actually get the photo from the mye28 storage. 

These scripts DO NOT RUN REGULARLY. They are intended to be run once or infrequently. I last ran them yesterday.

So any post created to an externally linked img tag after yesterday STILL comes from its original source.

This solves 2 problems:

- It ensures that externally linked photos will be safely archived and available on mye28 even if the external source (e.g. photofuckit, e28-535i.com, flickr) goes away (user deletes flickr account, etc...).

- It ensures photos linked to e28-535i.com and other ancient websites that do not not support HTTPS work, by creating a copy that IS available under HTTPS.

The MyE28 Photo Uploader
 
This does not yet exist. I am working on creating it.

PhpBB has a built in uploader called "Attachments" which sucks. We will not be using it.

Once the The MyE28 Photo Uploader is implemented the need for running the Legacy External Photo Links scripts on a regular basis will go away because most people will use the uploader for new posts. 
ahab
Posts: 6201
Joined: Jun 11, 2006 9:12 AM
Location: Chalfont, PA

Re: Photo hosting on MyE28

Post by ahab »

OK, thanks for the explanation (and of course the herculean effort behind it). 
Justin_FL
MyE28 IT Guru
MyE28 IT Guru
Posts: 2837
Joined: Feb 12, 2006 12:00 PM
Location: Palm Beach
Contact:

Re: Photo hosting on MyE28

Post by Justin_FL »

tig wrote: Jan 06, 2025 2:46 PMThis is not quite right. There are two scenarios:
Yeah, I figured that was the extension you were using or at least something similar from what I remember after looking into it years ago. Was just trying to confirm his general theory sounded about right without nerding out, so I appreciate the additional technical details!

Anyway, I spun up a Google Cloud bucket and uploaded my old website picture contents so hopefully my post images would be grabbed (until I figure out a more permanent host, etc after the trial period). Looks like at least some pics are cached already. I also edited a bunch of old posts to point to correct links/domains where they changed from like 20 years ago, so I'm sure they will be grabbed the next time you run the script.

What I noticed and not sure if it is specific to my domain/cloud config, but the rewrites here seem to point to files without an extension. Maybe it doesn't matter?

Example image:
https://mye28.z13.web.core.windows.net/ ... 0c051eeee3

From this post:
viewtopic.php?p=863581#p863581
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

What I noticed and not sure if it is specific to my domain/cloud config, but the rewrites here seem to point to files without an extension. Maybe it doesn't matter?

Example image:
https://mye28.z13.web.core.windows.net/ ... 0c051eeee3

From this post:
viewtopic.php?p=863581#p863581
That's a bug (I think).

Here's the links in that post:

Code: Select all

[img]http://www.jtdigital.com/gallery/pictures/5erFest/2003/JustinHowe/DSCN0700.jpg[/img]
The database says:

Image

But the HTML the extension is emitting is:

Code: Select all

<img src="https://mye28.z13.web.core.windows.net/external/3ca574b92784a30a624cc00c051eeee3" class="postimage" alt="Image">
An that aint right.

If you try to go to https://mye28.z13.web.core.windows.net/ ... 0c051eeee3 directly the browser tries to DL the image. But when in the site, it renders correctly.

So it probably doesn't matter, but I want to fix it because it smells wrong to my OCD.

Hmmm...
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »


So it probably doesn't matter, but I want to fix it because it smells wrong to my OCD.

Hmmm...
Fixed.
1st 5er
Posts: 21870
Joined: Jun 13, 2008 12:15 AM
Location: Cypress
Contact:

Re: Photo hosting on MyE28

Post by 1st 5er »

Worked for me. 
Thanks tig, 1st pic since Peter's host left. 

Image
tig
Posts: 9351
Joined: Mar 18, 2013 6:25 PM
Location: Durango
Contact:

Re: Photo hosting on MyE28

Post by tig »

I've updated the Forum FAQ for posting photos:

How to Post Photos, Pictures, and other Images - MyE28.com
Post Reply