File storage server

[pyrre]

I am about to set up a file server.
But.
I have specific needs.
I want to have disks in. sort of a "cluster".
I do not want to run ANY KIND of raid.
But i want the ability to add or remove disks as i see fit.
And the disks MUST be available in the same mountpoin/share...
Do any of you have any suggestion to what os and what software to use?

[X-ray]

"I do not want to run ANY KIND of raid.
But i want the ability to add or remove disks as i see fit."
------------------------------------------------------------

I don't know how you can remove disks as you see fit without having them in a RAID.

[pyrre]

Maybe you misunderstood... :-)

not hotswaping.
In a raid you need all disks to bee equal size.
And in raid 0 you have absolutely no data security. if one disk fails all data is lost.
In raid 5 you do not use all available space for data storage. Some space goes away in redundancy.

I just want to add several disks into one logical drive.
And the ability to add disks to make the storage space larger. or change disks as i buy newer end bigger ones.
And some sort of fail safe. if one disk dies only the data on that disk is lost, and the drive is replaceable.

Hope that clarifies my intentions...

[X-ray]

Oh I see, you don't mind losing the data on the one that dies.

[pyrre]

well... I do mind...
but over the past years i have been working on computers, only one disk has died of natural causes.
And it made "clicking" sounds for 6 months before it really died... Normally plenty of times to replace it.
However.
I cannot afford purchasing a large number of drives at once. and saving up to do so takes much too long time.
I just want the possibility to ad disks as i go along.
And of course replace failing disks...

Secure storage of data is already taken care of.
i just need a large storage for movies and tv series...
(most of wich is purchased, and i do not like "disk jockeying"  :-D )

[X-ray]

Okay, then get a RAID controller that supports JABOD (Just a Bunch of Disks).
You can have odd sizes, and you don't need drives on the controller for redundancy.
The trouble is, if you lose a drive, there is no neat way to get the data back in place.

[kevh100]

As X-ray said you really need some level of redundancy when dealing with multiple drives in a pool unless you:
a) are feeling lucky punk.
b) only have data which can be safely lost if you had a problem
or
c) keep good backups elsewhere

You don't need a raid controller that does JBOD though. You can use logical volumes on linux with LVM or ZFS on Solaris/OpenSolaris with any device that the OS can see using disks attached to any controller (e.g. some on IDE some on SATA and some on USB)

ZFS is very easy to use and get to grips with and offers many cool features (Filesystem compression, incremental snapshots, 256bit checksums on data).There are a few downsides, such as ZFS doesn't allow you to remove disks from a pool (yet). You can grow it using odd sized disks in any arrangement though. OpenSolaris works best on fairly modern hardware though and can need lots of RAM (>1GB) to get the best performance. Learning OpenSolaris is quite trivial and there is not too much of a learning curve. OpenSolaris doesn't support as much hardware as Linux either. You can get ZFS on linux but it's still has some issues.

I'd stick with LVM if I were you simply on the basis that you might possibly need to reduce the size of your data pool.

Kev

[pyrre]

The thought strike me of using Linux and LVM.
The computer to use as file server is a PIII 1000 with 768MB ram.

So far it has 8 disks of total 425 GIG.
Which will be replaced as i purchase bigger disks...
For starters i will test different OS and see what fits best.

according Linux.
How difficult is it to learn how to use Linux?
How may i reduce footprint of os and tweak the OS for my specific needs and hardware?

[Oliver]

Learning Linux is a widely various task. Have you had any background in Unix type systems? Will this be a dedicated fileserver? Will it be running on a wired NIC (some wireless interfaces are a PITA with Linux driver support)?

In terms of how streamlined your server could be, it also depends on what else you might need to do with the machine, and also if you need a GUI. Running a Linux kernel on your hardware shouldn't be any problem. You could quite possibly run a dedicated server installation, without any of the usual desktop/workstation features. It depends on what you are comfortable with, though. I think using a more fully featured distribution may be easier for you to come to terms with the system.

You could try learning a bit about Linux with a virtual installation somewhere, or with a live CD/DVD on your server hardware, just to get used to it for a while. This is a text which I found handy when first learning to use Linux based systems. From memory, I don't think it goes into running a file server. However, it's not bad for getting started.

[pyrre]

It will mainly be running a file server.
hopefully with a HTTP interface.
if the server runs gui or not does not really matter.
Linux based "openfiler" is a probability.

I have absolutely no previous linux or unix experience.

The server setup:
Intel PIII 1000 CPU
Via TUV4X motherboard
768MB ram
silicon image PCI IDE controller (not raid)
NVidia GF TI4200 (the smallest i had available)
CNet gigabit wired Ethernet adapter.
Disks:
10 gig fujitsu (system)
13,5 gig IBM
20 gig Maxtor
40 gig WD
40 gig WD
80 gig WD
120 gig WD
120 gig WD

All the disks will be used for testing and learning the os..
and will be replaced later on as economy seems fit... :-)

[persia]

Why serve files via http?  Wouldn't SMB be a better choice?

[pyrre]

SMB will be for sharing the files in the network, yes.
The GUI for configuring and using the server itself will be run on HTTP

[Trev]

Under Windows 2000+, you can mount drives as directories instead of drive letters. That's one simple way of adding storage. I posted something on how to arrange a Windows file system in a way similar to a standard Unix layout in a very old post. [Searching...] This is rather old and perhaps full or errors, but I'm not going to edit it. :-) Enjoy, though.

You could follow a scheme similar to Glaucus' but take better advantage of NTFS:

System Volume (Mounted as C:)
Application Volume (Mounted as C:\Program FIles)

Data and temp stuff gets tricky, 'cause most people just use the defaults. On Windows XP, the temp and personal folders are located here:

C:\Documents and Settings\%USERNAME%\Local Settings\Temp
C:\Documents and Settings\%USERNAME%\My Documents

(Really, it's %USERPROFILE%\..., but %SystemDrive%\Documents and Settings is the default location for storing profiles.)

Personally, I just format the whole drive as a single partition and use directories to organize stuff. A good compromise might be setting up two volumes:

System Volume (C:)
Data Volume (E:)

And when your data volume gets full, you can just throw in another drive and append the new space to E:. The key is to think logically and not get stuck on how things are layed out physically. With a good, *ahem*, backup routine, it really shouldn't matter.

Trev

***

Mounting volumes as a folder (or both a folder and a drive letter or as a combination of multiple folders and drive letters) is a feature of NTFS, so all your volumes should be formatted as NTFS. Before you begin, decide how much space you want to use for each volume.

In this example:

System Volume: 20 GB
Application Volume: 80 GB

1. Install Windows XP Professional as per usual, manually creating a 20 GB partition for the system volume. Quick-format this partition as NTFS.

2. After the installation is complete, copy "C:\Program Files" to a new folder on the same drive. We'll use the default name of "C:\Copy of Program Files" for this example.

The "Program Files" folder is a normal folder; however, Windows Explorer prevents this folder from being renamed. To facilitate our change, we'll need to make two temporary registry changes.

3. Run regedit.exe and change the following values (you can copy and paste this into a .reg file if you'd like):

REGEDIT4
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
"CommonFilesDir"="C:\Copy of Program Files\Common Files"
"ProgramFilesDir"="C:\Copy of Program Files"

4. Reboot.

You may notice some user interface weirdness at this point. Don't worry. It will be resolved when the change is complete.

5. Delete the contents of "C:\Program Files". You should have an empty folder at this point.

6. Start the Computer Management MMC snap-in (Start | Settings | Control Panel | Administrative Tools | Computer Management).

7. Select the Storage | Disk Management node in the tree view.

In our example, we're going to use 80 GB of free space on our system drive to create a new volume. Some of the following text is paraphrased from the Disk Management help file.

8. Right-click on the unallocated region and select New Partition on the context menu.

9. In the New Partition Wizard, click Next, click Primary Partition. When prompted for the partition size, enter 80 GB, and then follow the instructions on your screen.

10. Right-click on the new partition and select Format from the context menu.

11. Format the new partition using the NTFS file system.

12. Right-click the formatted partition and select Change Drive Letter and Paths from the context menu.

13. If there's a drive letter listed, highlight it and click Remove.

14. Click Add. Click Mount in the following empty NTFS folder, and type "C:\Program Files" in the text box. Click OK.

The new partition is now mounted under "C:\Program Files".

15. Copy the contents of "C:\Copy of Program Files" to "C:\Program Files". Keep "C:\Copy of Program Files" around for later use.

16. Run regedit.exe and change the following values (you can copy and paste this into a .reg file if you'd like):

REGEDIT4
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
"CommonFilesDir"="C:\Program Files\Common Files"
"ProgramFilesDir"="C:\Program Files"

17. Reboot.

Voila! You should be mostly done. At this point, you should verify that the permissions on all files and folders under "C:\Program Files" match the original permissions under "C:\Copy of Program Files". You can delete the copy when you're done.

I haven't actually tested these instructions, but theoretically, they should work. Let me know if you have any problems. . . .

Trev

***

Quote:
--------------------------------------------------------------------------------

I assume you do the same for My Documents.
--------------------------------------------------------------------------------


I probably wouldn't do this for My Documents, but I might do it for "C:\Documents and Settings".

As a comparison to Un*x, you could make the following associations:

/ - "%SystemDrive%"
/bin - "%SystemRoot%\System32"
/etc- "%SystemRoot%\System32\Drivers\etc"
/home - "C:\Documents and Settings"
/opt - "C:\Program Files"
/sbin - "%SystemRoot%\System32"
/usr - "C:\Program Files"
/var - "%SystemRoot%\Temp"

Using those associations, you could use the procedure above to partition your file system in a manner similar to the way you would do it in Un*x. The only exception is %temp%, which varies from user to user. But in general, user temp stuff should go in the user's profile, and system temp stuff (specifically, temp file created by the LocalSystem account) should go in "%SystemRoot%\Temp".

Quote:
--------------------------------------------------------------------------------

If at a later time you add a second hard drive, could you mount it to the same directory, to increase the size of the mounted directory (for example, add another 40 gigs to c:\Program Files)
--------------------------------------------------------------------------------


Absolutely. In NTFS V5 lingo, this is called a spanned volume. The bad news is that you need to convert your basic disks to dynamic disks and this isn't supported by Windows XP Home. Furthermore, only Windows 2000, Windows XP Professional, and Windows 2003 can see dynamic volumes. (You could try Sysinternals' NTFS for Windows 98. NTFSDOS Professional should work as well.)

Take a look in the Disk Management help under the following topics (you'll have to remove the "http://" for these links to work):

Change a basic disk into a dynamic disk
Manage spanned volumes

You can only span a volume 32 times, and you can't expand a volume that has been striped or mirrored using the operating system's software. In the server world, we only use spanned volumes on hardware RAID (either mirrored or with parity--not RAID 0) arrays that are routinely backed up.

Trev

***

Quote:
--------------------------------------------------------------------------------

That is actually the coolest Windows tip I've heard for a very long time. I didn't know Windows allowed one to mount volumes at arbitrary locations, that was what I believed to be one of Linux's major advantages over it. So much for that I guess Since when has that been possibe? NT4, Win2K or WinXP?
--------------------------------------------------------------------------------


Windows 2000 Professional was the first OS to support it directly from the user interface. These are called Volume Mount Points, and they're supported by NTFS v5 and later. They're implemented in the OS using reparse points, which have been available for some time. Reparse points also support hard links and junctions at the file system level. (These aren't the same as shortcuts, which are just IShellLink objects persisted to disk.)

Behind the scenes, Windows NT has a fully hierarhcical object namespace similar to Un*x's file system space. Disk volumes are a very small part of that namespace. For example, "C:" is actually "\??\C:", which is typically a symbolic link to "\Device\HarddiskVolume1".

If you take away all the Microsoft marketing BS, Windows is actually a pretty spiffy OS. And really, the engineers that work for Microsoft are friendly, helpful people (who sometimes like to give away t-shirts and Microsoft video games).

Trev

***

If you're only dealing with Windows XP Professional (and frankly, unless you're using some kind of proprietary or turn-key solution or doing system administration or OS-specific testing and development, there's no reason to be using another version of Windows), then the drawbacks (what are they again?) of dynamic volumes shouldn't make a difference. The Logical Disk Manager subsystem was designed by Veritas and licensed by Microsoft, and it's definitely a more robust solution than PC BIOS based partitioning (although the system partition retains an entry in the partition table after the disk has been converted).

Now, about spanned volumes. The tagged on volumes (for lack of a better phrase) are not treated as separate entities with respect to volume mount points. In other words, you can't assign a separate drive letter to the additional space. When a volume is spanned across multiple devices, the space is used in the order in which the volumes were added, so disk0 will be filled up before disk1. The only safe way to remove a disk from a spanned volume is to back up the entire volume, delete it, remove the disk, recreate the volume, and restore the data. So there are some drawbacks there.

The only reason you wouldn't want to convert a basic disk to a dynamic disk is if you're going to access that disk from other operating systems. If you're only using NTFS partitions, then the point is moot anyway, as you can't normally access NTFS partitions from other operating systems.

So, after you've backed up your data, go for it!

EDIT: Can we change the displayed topic of this thread?

EDIT2: I definitely don't mind the questions. It's nice to feel useful.

[persia]

Yeah, best to can the GUI.  I run a small filesever using Gentoo Linux.  It's straight forward and easy to maintain, of course there's no redundancy and no protection against failure, but it works well if you only have a few computers accessing the files and you have some other method of backup.

I'm not sure what you need to change, file servers in the home are a set 'em and forget 'em technology.

 

[Trev]

If you don't want to run Windows, don't want to configure the latest version of Samba, and don't care whether or not your file server is 100% SMB-compatible or performs well in a high-bandwidth, low-latency environment (most consumer NAS products do not), then just invest $150-$300 in an off-the-shelf NAS solution from Iomega, Seagate, or some other vendor. But, you probably want other nice features like UPnP, mDNS (Bonjour), etc.